Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In order to quantitatively analyze such an intriguing phenomenon, we propose a concept alignment method based on how units respond to the replicated text.

The evidence to be evaluated is:Compared with these previous studies, we focus on discovering a much wider variety of _natural language concepts_, including any morphemes, words, and phrases all found in the training data. To the best our knowledge, this is the first attempt to discover concepts among all that exist in the form of natural language from the training corpus. By extending the scope of detected concepts to meaningful building blocks of natural language, we provide insights into how various linguistic features are encoded by the hidden units of deep representations.

(1) By identifying the role of the individual units, rather than analyzing the representation as a whole, we provide more fine-grained understanding of how the representations encode informative factors in training data.

(3) Our concept alignment method does not need any additional labeled data or re-training process, so it can always provide deterministic interpretation results using only the training data.

Our approach for aligning natural language concepts to units is summarized as follows. We first train a CNN model for each natural language task (_e.g._ translation and classification) and retrieve training sentences that highly activate specific units. Interestingly, we discover morphemes, words and phrases that appear dominantly within these retrieved sentences, implying that those concepts have a significant impact on the activation value of the unit. Then, we find a set of concepts which attribute a lot to the unit activation by measuring activation value of each replicated candidate concept, and align them to unit.

Given a layer and sentence \(s\in\mathcal{S}\), let \(\tilde{A}^{t}_{u}(s)\)denote the activation of unit \(u\) at spatial location \(l\). Then, for unit \(u\), we average activations over all spatial locations as \(a_{u}(s)=\frac{1}{Z}\sum_{l}A_{u}^{l}(s)\), where \(Z\) is a normalizer. We then retrieve top \(K\) training sentences per unit with the highest mean activation \(a_{u}\).

We propose a simple approach for identifying the concepts as follows. For constructing candidate concepts, we parse each of top \(K\) sentences with a constituency parser (Kitaev and Klein, 2018). Within the constituency-based parse tree, we define candidate concepts as all terminal and non-terminal nodes (_e.g._ from sentence _John hit the balls_, we obtain candidate concepts as {_John, hit, the balls, hit the balls, John hit the balls_}). We also break each word into morphemes using a morphological analysis tool (Virpioja et al., 2013) and add them to candidate concepts (_e.g._ from word _balls_, we obtain morphemes {_ball, s_}). We repeat this process for every top \(K\) sentence and build a set of candidate concepts for unit \(u\), which is denoted as \(\mathcal{C}_{u}=\{c_{1},...,c_{N}\}\), where \(N\) is the number of candidate concepts of the unit. Next, we measure how each candidate concept contributes to the unit's activation value. For normalizing the degree of an input signal to the unit activation, we create a synthetic sentence by replicating each candidate concept so that its length is identical to the average length of all training sentences (_e.g._ candidate concept _the ball_ is replicated as _the ball the ball the ball_...). Replicated sentences are denoted as \(\mathcal{R}=\{r_{1},...,r_{N}\}\), and each \(r_{n}\in\mathcal{R}\) is forwarded to CNN, and their activation value of unit \(u\) is measured as \(a_{u}(r_{n})\in\mathbb{R}\), which is averaged over \(l\) entries. Finally, the degree of alignment (DoA) between a candidate concept \(c_{n}\) and a unit \(u\) is defined as follows: \[\mathsf{DoA}_{u,c_{n}}=a_{u}(r_{n}) \tag{1}\] In short, the \(\mathsf{DoA}\)1 measures the extent to unit \(u\)'s activation is sensitive to the presence of candidate concept \(c_{n}\). If a candidate concept \(c_{n}\) appears in the top \(K\) sentences and unit's activation value \(a_{u}\) is responsive to \(c_{n}\) a lot, then \(\mathsf{DoA}_{u,c_{n}}\) gets large, suggesting that candidate concept \(c_{n}\) is strongly aligned to unit \(u\).

Finally, for each unit \(u\), we define a set of its aligned concepts \(\mathcal{C}_{u}^{*}=\{c_{1}^{*},...,c_{M}^{*}\}\) as \(M\) candidate concepts with the largest DoA values in \(C_{u}\). Depending on how we set \(M\), we can detect different numbers of concepts per unit.

To quantitatively evaluate how well our approach aligns concepts, we measure how selectively each unit responds to the aligned concept. Motivated by Morcos et al. (2018), we define the **concept selectivity** of a unit \(u\), to a set of concepts \(\mathcal{C}_{u}^{*}\) that our alignment method detects, as follows: \[\mathsf{Sel}_{u}=\frac{\mu_{+}-\mu_{-}}{\max_{s\in\mathcal{S}}a_{u}(s)-\min_{s \in\mathcal{S}}a_{u}(s)} \tag{2}\] where \(\mathcal{S}\) denotes all sentences in training set, and \(\mu_{+}=\frac{1}{|\mathcal{S}_{+}|}\sum_{s\in\mathcal{S}_{+}}a_{u}(s)\) is the average value of unit activation when forwarding a set of sentences \(\mathcal{S}_{+}\), which is defined as one of the following: * _replicate_: \(\mathcal{S}_{+}\) contains the sentences created by replicating each concept in \(\mathcal{C}_{u}^{*}\). As before, the sentence length is set as the average length of all training sentences for fair comparison.

In contrast, \(\mu_{-}=\frac{1}{|\mathcal{S}_{-}|}\sum_{s\in\mathcal{S}_{-}}a_{u}(s)\) is the average value of unit activation when forwarding \(\mathcal{S}_{-}\), which consists of training sentences that do _not_ include any concept in \(\mathcal{C}_{u}^{*}\). Intuitively, if unit \(u\)'s activation is highly sensitive to \(\mathcal{C}_{u}^{*}\) (_i.e._ those found by our alignment method) and if it is not to other factors, then \(\mathsf{Sel}_{u}\) gets large; otherwise, \(\mathsf{Sel}_{u}\) is near 0. More interestingly, we discover that many units could capture specific meanings or syntactic roles beyond superficial, low-level patterns. For example, unit 690 of the 14th layer in ByteNet captures (_what, who, where_) concepts, all of which play a similar grammatical role. On the other hand, unit 224 of the 14th layer in ByteNet and unit 53 of the 0th layer in VDCNN each captures semantically similar concepts, with the ByteNet unit detecting the meaning of certainty in knowledge (_sure, know, aware_) and the VDCNN unit detecting years (_1999, 1969, 1992_).

We note that there are units that detect concepts more abstract than just morphemes, words, or phrases, and for these units, our method tends to align relevant lower-level concepts. For example, in unit 244 of the 3rd layer in VDCNN, while each aligned concept emerges only once in the top \(K\) sentences, all top \(K\) sentences have similar _nucances_ like positive sentiments. In this case, our method does capture relevant phrase-level concepts (e.g., _very disappointing, absolute worst place_), indicating that the higher-level _nucance_ (e.g., negativity) is indirectly captured.

Our approach thus tends to find concepts that are frequent in training data or considered as more important than others for solving the target task.

* Several units in the CNN learned on NLP tasks respond selectively to specific natural language concepts, rather than getting activated in an uninterpretable way. This means that these units can serve as detectors for specific natural language concepts.

* There are units capturing syntactically or semantically related concepts, suggesting that they model the _meaning or grammatical role shared between those concepts_, as opposed to superficially modeling each natural language _symbol_.

Overall, we find that data and task-specific concepts are likely to be aligned to many units. In AG News, since the task is to classify given sentences into following categories; _World_, _Sports_, _Business_ and _Science/Tech_, concepts related to these topics commonly emerge. Similarly, we can see that units learned for Europarl dataset focus to encode some key words (_e.g. vote_, _propose_, _environment_) in the training corpus.

To confirm that it also holds for representations learned in NLP tasks, we divide granularity of natural language concepts to the morpheme, word and \(N\)-gram phrase (\(N=2,3,4,5\)), and observe the number of units that they are aligned in different layers.

To validate this hypothesis, we re-train ByteNet from scratch while varying only layer depth of the encoder and fixing other conditions. We record their BLEU scores on the validation data as shown in Figure 6. The performance of the translation model does not change much with more than six encoder layers, but it significantly drops at the models with fewer than 4 encoder layers. This trend coincides with the result from Figure 5 that the evolution of concept granularity stops around middle-to-higher layers. This shared pattern suggests that about six encoder layers are enough to encode informative factors in the given datasets to perform optimally on the translation task. In deeper models, this may suggest that the middle layer's representation may be already informative enough to encode the input text, and our result may partly coincide with that of Mou et al. (2016), which shows that representation of intermediate layers is more transferable than that of deeper layers in language tasks, unlike in computer vision where deeper layers are usually more useful and discriminative.

We observe that the concepts do not appear uniformly; some concepts are aligned to many units, while others are aligned to few or even no units.

_Two possible hypotheses may explain the emergence of dominant concepts. First, the concepts with a higher frequency in training data may be aligned to more units.

Second, the concepts that have more influence on the objective function (expected loss) may be aligned to more units. We can measure the effect of concept \(c\) on the task performance as _Delta of Expected Loss_ (DEL) as follows: \[\mathsf{DEL}(c)=\mathbb{E}_{s\in\mathcal{S},y\in\mathcal{Y}}[\mathcal{L}(s,y) ]-\mathbb{E}_{s\in\mathcal{S},y\in\mathcal{Y}}[\mathcal{L}(\mathsf{Occ}_{c}(s ),y)] \tag{3}\]

As a result, \(\mathsf{DEL}(c)\) measures the impact of concept \(c\) on the loss function, where a large positive value implies that concept \(c\) has an important role for solving the target task.

We proposed a simple but highly effective concept alignment method for character-level CNNs to confirm that each unit of the hidden layers serves as detectors of natural language concepts. Using this method, we analyzed the characteristics of units with multiple datasets on classification and translation tasks.

Another direction is to quantify the properties of individual units in other models widely used in NLP tasks.

In section 3.2, we define Degree of Alignment (DoA) between concept \(c_{n}\) and unit \(u\) as activation value of unit \(u\) for replication of \(c_{n}\).

We then define DoA between candidate concept \(c_{n}\) and unit \(u\) by using PMI as follow: \[\text{DoA}_{u,c_{n}}=\text{pmi}(u,c_{n})=\log\frac{p(u,c_{n})}{p(u)p(c_{n})}, \text{where} \tag{5a}\] \[p(u)=\frac{\#[s\in\mathcal{S},s\in\text{topK}(u)]}{\#[s\in \mathcal{S}]},\] (5b) \[p(c_{n})=\frac{\#[s\in\mathcal{S},c_{n}\in s]}{\#[s\in \mathcal{S}]},\] (5c) \[p(u,c_{n})=\frac{\#[s\in\text{topK}(u),c_{n}\in s]}{\#[s\in \text{topK}(u)]} \tag{5d}\]

We tested concept alignments with the following concept occlusion method. For each of the top \(K\) sentences, we replace a \(c_{n}\) by dummy character tokens which have no meaning, forward it to the model, and measure the reduction of the unit activation value. We repeat this for every candidate concept in the sentences - as a result, we can identify which candidate concept greatly reduce unit activation values. We thus define concepts aligned to each unit as the candidate concept that consistently lower the unit activation across the top \(K\) sentences. More formally, for each unit \(u\), let \(\mathcal{S}=\{s_{1},...,s_{K}\}\) be top \(K\) activated sentences. Since we occlude each candidate concept in sentences, we define the set of candidate concept \(\mathcal{C}=\{c_{1},...,c_{N}\}\), obtained from parsing each sentence in \(\mathcal{S}\). We define the **degree of alignment (DoA)** between a concept \(c\in\mathcal{C}\) and a unit \(u\) as: \[\text{DoA}_{u,c_{n}}=\frac{1}{Z}\sum_{s\in\mathcal{S}}\mathbb{1}(c_{n}\in s)( a_{u}(s)-a_{u}(\text{Occ}_{c_{n}}(s))) \tag{6}\] where \(Z\) is a normalizing factor, and \(a_{u}\) indicates the mean activation of unit \(u\), \(\text{Occ}_{c_{n}}(s)\) is a sentence \(s\) where candidate concept \(c_{n}\) is occluded, and \(\mathbb{1}(c_{n}\in s)\) is an indicator of whether \(c_{n}\) is included in the sentence \(s\). In short, the DoA measures how much a candidate concept contributes to the activation of the unit's top \(K\) sentences. If a candidate concept \(c_{n}\) appears in the topsentences \(\mathcal{S}\) and greatly reduces the activation of unit \(u\), then \(\mathsf{DoA}_{u,c_{n}}\) gets large, implying that the \(c_{n}\) is strongly aligned to unit \(u\).

In Section 3.2, we set \(M=3\). Although \(M\) is used as a threshold to set how many concepts per unit are considered, different \(M\) values have little influence on quantitative results such as selectivity in Section 4.2. Figure 8 shows the mean and variance of selectivity values with different \(M=[1,3,5,10]\), where there is little variants in the overall trend; the sensitivity of the _replicate_ set is the highest, and that of _one instance_ is runner-up, and that of _random_ is the lowest.

We first define the unit interpretability for unit \(u\) as follows: \[\mathsf{Interpretability}(u)=\begin{cases}1&\text{if max}_{s\in\mathcal{S}}\{a _{u}(s)\}<\text{max}_{i=1,\dots,N}\{r_{i}\}\\ 0&\text{otherwise}\end{cases} \tag{7}\] where \(\mathcal{S}\) is the set of training sentences, \(a_{u}(s)\) is the activation value of unit \(u\), and \(r_{i}\) is the activation value of the sentence that is made up of replicating concept \(c_{i}\). We define unit \(u\) as interpretable when its \(\mathsf{Interpretability}(u)\) equals to 1, and otherwise as non-interpretable. The intuition is that if a replicated sentence that is composed of only one concept has a less activation value than the top-activated sentences, the unit is not sensitive to the concept compared to a sequence of different words.

We introduce some units whose concepts have the shared meaning in Section 4.3. We here refer _concept cluster_ to the concepts that are aligned to the same unit and have similar semantics or grammatical roles. We analyze how clusters are formed in the units and how they vary with the target task and layer depth. Each central heat map represents the number of times each concept pair is aligned to the same unit. Since the concepts in the x, y-axes are ordered by the clustering result, if the diagonal blocks (concept clusters) emerge more strongly, the concepts in the same unit are more likely to have the similar meanings.

We analyze how concept clusters change by layer in each task. We compute the averaged pairwise distance between the concepts in each layer. We project each concept to the vector space using the three pretrained embeddings: (1) Glove (Pennington et al., 2014), (2) ConceptNet (Speer et al., 2017), (3) fastText.

Consistent with our intuition, in all datasets, both the document frequency and the delta of expected loss are closely related to the number of units per concept. It concludes that the representations are learned for identifying not only the frequent concepts in the training set and but also the important concepts for solving the target task.

In the classification tasks, we expect to find more concepts that are directly related to predicting the output label, as opposed to the translation tasks where the representations may have to include information on most of the words for an accurate translation. While our goal is not to relate each concept to one of the labels, we find several concepts that are more predictive to a particular label than others. Consistent with section 4.4, there are data-specific and task-specific concepts aligned in each layer; _i.e._ {_worst, 2 stars, awful_} at Yelp Review, {_film, ship, school_} at DBpedia, and some key words at translation datasets.

Input: Figure 8: Mean and variance of selectivity values for different M = [1, 3, 5, 10], where M is the number of selected concepts per unit. In all settings, the selectivity of replicate ones is the highest, that of one instance ones is runner-up, and that of random is the lowest near 0.

Input: Figure 6: BLEU scores on the validation data for three translation models. We train ByteNet from scratch on each dataset by varying the number of encoder layers.

Input: Figure 4: 30 concepts selected by the number of aligned units in three encoding layers of ByteNet learned on the Europarl translation dataset (left column) and VDCNN learned on AG-News (right column). [#] symbol denotes morpheme concepts. See section 4.4 for details.

Input: Figure 3: Examples of top activated sentences and aligned concepts to some units in several encoding layers of ByteNet and VDCNN. For each unit, concepts and their presence in top K sentences are shown in the same color. [#] symbol denotes morpheme concepts. See section 4.3 for details.

Input: Figure 10: Examples of non-interpretable units, their concepts and top 5 activated sentences. Units are from representations learned on english-to-french translation dataset. See section D for details.

Input: Figure 10: Examples of non-interpretable units, their concepts and top 5 activated sentences. Units are from representations learned on english-to-french translation dataset. See section D for details.

Input: Figure 1: We discover the most activated sentences and aligned concepts to the units in hidden representations of deep convolutional networks. Aligned concepts appear frequently in most activated sentences, implying that those units respond selectively to specific natural language concepts.

Input: Figure 14: The Pearson correlations between the number of units per concept and (i) document frequency (top row), (ii) delta of expected loss (bottom row). They are measured at the final layer representation.

Input: Figure 15: 30 concepts selected by the number of aligned units in four encoding layers in VDCNN learned on Yelp Review dataset and DBpedia ontology dataset, and ByteNet learned on the English-to-German, English-to-French, and English-to-Czech parallel corpus. [#] symbol denotes morpheme concept.

Input: Figure 12: Averaged pairwise distances of concepts in each layer per task. Top and bottom row show the concept distances of translation models and classification models, respectively. For projecting concepts into the embedding space, we use three pretrained embedding: Glove, ConceptNet, FastText. See Appendix E.2 for more details.

Input: Figure 13: The number of unique concepts in each layer. It increases with the layer depth, which implies that the units in a deeper layer represent more diverse concepts.

Input: Figure 16: Aligned concepts per each task and their number of occurrences over multiple layers. See Appendix H for more details.

Input: Figure 11: Concept clusters of the last layer representations learned in each task. The more distinct the diagonal blocks are, the stronger the tendency that concepts aligned to the same unit share similar meanings or semantics. See Appendix E for details.

Input: Figure 14: The Pearson correlations between the number of units per concept and (i) document frequency (top row), (ii) delta of expected loss (bottom row). They are measured at the final layer representation.

Input: Figure 7: Correlations between the number of units per concept and (a) document frequency and (b) delta of expected loss. Pearson correlation coefficients are measured at the final layer of the AG News sentence classification task. Some concepts aligned to many units are annotated. Results on other tasks are available in Appendix F.

Input: Figure 15: 30 concepts selected by the number of aligned units in four encoding layers in VDCNN learned on Yelp Review dataset and DBpedia ontology dataset, and ByteNet learned on the English-to-German, English-to-French, and English-to-Czech parallel corpus. [#] symbol denotes morpheme concept.

Input: Figure 16: Aligned concepts per each task and their number of occurrences over multiple layers. See Appendix H for more details.

Input: Figure 2: Mean and variance of selectivity values over all units in the learned representation for each dataset. Sentences including the concepts that our alignment method discovers always activate units significantly more than random sentences. See section 4.2 for details.

Input: Figure 11: Concept clusters of the last layer representations learned in each task. The more distinct the diagonal blocks are, the stronger the tendency that concepts aligned to the same unit share similar meanings or semantics. See Appendix E for details.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim about proposing a concept alignment method based on unit responses to replicated text is well-stated and accurately reflects what is demonstrated in the evidence. The paper provides detailed mathematical formulations (equations 1-3, 5-7) explaining how the concept alignment works through measuring unit activations to replicated text. The figures show empirical validation through selectivity analysis (Fig 2,8), correlation studies (Fig 7,14), and qualitative examples (Fig 1,3). The method is clearly defined and supported by comprehensive experimental results without exaggeration.

<score>1</score>

Justification:
- The claim directly aligns with the detailed technical description of the method in the evidence
- Multiple evaluation approaches validate the concept alignment approach
- Key equations explicitly show how unit responses to replicated text are used
- Results demonstrate the method works as claimed without overstating capabilities
- No grandiose or unsupported claims are made about the method's capabilities


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In order to quantitatively analyze such an intriguing phenomenon, we propose a concept alignment method based on how units respond to the replicated text.

The review comment to be evaluated is:

The paper is well written and structured, presenting the problem clearly and accurately. It contains considerable relevant references and enough background knowledge. It nicely motivates the proposed approach, locates the contributions in the state-of-the-art and reviews related work. It is also very honest in terms of how it differs on the technical level from existing approaches.
The paper presents interesting and novel findings to further state-of-the-art’s understanding on how language concepts are represented in the intermediate layers of deep convolutional neural networks, showing that channels in convolutional representations are selectively sensitive to specific natural language concepts. It also nicely discusses how concepts granularity evolves with layers’ deepness in the case of natural language tasks.
What I am missing, however, is an empirical study of concepts coverage over multiple layers, studying the multiple occurrences of single concepts at different layers, and a deeper dive on the rather noisy elements of natural language and the layers’ activation dynamics towards such elements.
Overall, however, the ideas presented in the paper are interesting and original, and the experimental section is convincing. My recommendation is to accept this submission.

The evidence to be evaluated is:Compared with these previous studies, we focus on discovering a much wider variety of _natural language concepts_, including any morphemes, words, and phrases all found in the training data. To the best our knowledge, this is the first attempt to discover concepts among all that exist in the form of natural language from the training corpus. By extending the scope of detected concepts to meaningful building blocks of natural language, we provide insights into how various linguistic features are encoded by the hidden units of deep representations.

(1) By identifying the role of the individual units, rather than analyzing the representation as a whole, we provide more fine-grained understanding of how the representations encode informative factors in training data.

(3) Our concept alignment method does not need any additional labeled data or re-training process, so it can always provide deterministic interpretation results using only the training data.

Our approach for aligning natural language concepts to units is summarized as follows. We first train a CNN model for each natural language task (_e.g._ translation and classification) and retrieve training sentences that highly activate specific units. Interestingly, we discover morphemes, words and phrases that appear dominantly within these retrieved sentences, implying that those concepts have a significant impact on the activation value of the unit. Then, we find a set of concepts which attribute a lot to the unit activation by measuring activation value of each replicated candidate concept, and align them to unit.

Given a layer and sentence \(s\in\mathcal{S}\), let \(\tilde{A}^{t}_{u}(s)\)denote the activation of unit \(u\) at spatial location \(l\). Then, for unit \(u\), we average activations over all spatial locations as \(a_{u}(s)=\frac{1}{Z}\sum_{l}A_{u}^{l}(s)\), where \(Z\) is a normalizer. We then retrieve top \(K\) training sentences per unit with the highest mean activation \(a_{u}\).

We propose a simple approach for identifying the concepts as follows. For constructing candidate concepts, we parse each of top \(K\) sentences with a constituency parser (Kitaev and Klein, 2018). Within the constituency-based parse tree, we define candidate concepts as all terminal and non-terminal nodes (_e.g._ from sentence _John hit the balls_, we obtain candidate concepts as {_John, hit, the balls, hit the balls, John hit the balls_}). We also break each word into morphemes using a morphological analysis tool (Virpioja et al., 2013) and add them to candidate concepts (_e.g._ from word _balls_, we obtain morphemes {_ball, s_}). We repeat this process for every top \(K\) sentence and build a set of candidate concepts for unit \(u\), which is denoted as \(\mathcal{C}_{u}=\{c_{1},...,c_{N}\}\), where \(N\) is the number of candidate concepts of the unit. Next, we measure how each candidate concept contributes to the unit's activation value. For normalizing the degree of an input signal to the unit activation, we create a synthetic sentence by replicating each candidate concept so that its length is identical to the average length of all training sentences (_e.g._ candidate concept _the ball_ is replicated as _the ball the ball the ball_...). Replicated sentences are denoted as \(\mathcal{R}=\{r_{1},...,r_{N}\}\), and each \(r_{n}\in\mathcal{R}\) is forwarded to CNN, and their activation value of unit \(u\) is measured as \(a_{u}(r_{n})\in\mathbb{R}\), which is averaged over \(l\) entries. Finally, the degree of alignment (DoA) between a candidate concept \(c_{n}\) and a unit \(u\) is defined as follows: \[\mathsf{DoA}_{u,c_{n}}=a_{u}(r_{n}) \tag{1}\] In short, the \(\mathsf{DoA}\)1 measures the extent to unit \(u\)'s activation is sensitive to the presence of candidate concept \(c_{n}\). If a candidate concept \(c_{n}\) appears in the top \(K\) sentences and unit's activation value \(a_{u}\) is responsive to \(c_{n}\) a lot, then \(\mathsf{DoA}_{u,c_{n}}\) gets large, suggesting that candidate concept \(c_{n}\) is strongly aligned to unit \(u\).

Finally, for each unit \(u\), we define a set of its aligned concepts \(\mathcal{C}_{u}^{*}=\{c_{1}^{*},...,c_{M}^{*}\}\) as \(M\) candidate concepts with the largest DoA values in \(C_{u}\). Depending on how we set \(M\), we can detect different numbers of concepts per unit.

To quantitatively evaluate how well our approach aligns concepts, we measure how selectively each unit responds to the aligned concept. Motivated by Morcos et al. (2018), we define the **concept selectivity** of a unit \(u\), to a set of concepts \(\mathcal{C}_{u}^{*}\) that our alignment method detects, as follows: \[\mathsf{Sel}_{u}=\frac{\mu_{+}-\mu_{-}}{\max_{s\in\mathcal{S}}a_{u}(s)-\min_{s \in\mathcal{S}}a_{u}(s)} \tag{2}\] where \(\mathcal{S}\) denotes all sentences in training set, and \(\mu_{+}=\frac{1}{|\mathcal{S}_{+}|}\sum_{s\in\mathcal{S}_{+}}a_{u}(s)\) is the average value of unit activation when forwarding a set of sentences \(\mathcal{S}_{+}\), which is defined as one of the following: * _replicate_: \(\mathcal{S}_{+}\) contains the sentences created by replicating each concept in \(\mathcal{C}_{u}^{*}\). As before, the sentence length is set as the average length of all training sentences for fair comparison.

In contrast, \(\mu_{-}=\frac{1}{|\mathcal{S}_{-}|}\sum_{s\in\mathcal{S}_{-}}a_{u}(s)\) is the average value of unit activation when forwarding \(\mathcal{S}_{-}\), which consists of training sentences that do _not_ include any concept in \(\mathcal{C}_{u}^{*}\). Intuitively, if unit \(u\)'s activation is highly sensitive to \(\mathcal{C}_{u}^{*}\) (_i.e._ those found by our alignment method) and if it is not to other factors, then \(\mathsf{Sel}_{u}\) gets large; otherwise, \(\mathsf{Sel}_{u}\) is near 0. More interestingly, we discover that many units could capture specific meanings or syntactic roles beyond superficial, low-level patterns. For example, unit 690 of the 14th layer in ByteNet captures (_what, who, where_) concepts, all of which play a similar grammatical role. On the other hand, unit 224 of the 14th layer in ByteNet and unit 53 of the 0th layer in VDCNN each captures semantically similar concepts, with the ByteNet unit detecting the meaning of certainty in knowledge (_sure, know, aware_) and the VDCNN unit detecting years (_1999, 1969, 1992_).

We note that there are units that detect concepts more abstract than just morphemes, words, or phrases, and for these units, our method tends to align relevant lower-level concepts. For example, in unit 244 of the 3rd layer in VDCNN, while each aligned concept emerges only once in the top \(K\) sentences, all top \(K\) sentences have similar _nucances_ like positive sentiments. In this case, our method does capture relevant phrase-level concepts (e.g., _very disappointing, absolute worst place_), indicating that the higher-level _nucance_ (e.g., negativity) is indirectly captured.

Our approach thus tends to find concepts that are frequent in training data or considered as more important than others for solving the target task.

* Several units in the CNN learned on NLP tasks respond selectively to specific natural language concepts, rather than getting activated in an uninterpretable way. This means that these units can serve as detectors for specific natural language concepts.

* There are units capturing syntactically or semantically related concepts, suggesting that they model the _meaning or grammatical role shared between those concepts_, as opposed to superficially modeling each natural language _symbol_.

Overall, we find that data and task-specific concepts are likely to be aligned to many units. In AG News, since the task is to classify given sentences into following categories; _World_, _Sports_, _Business_ and _Science/Tech_, concepts related to these topics commonly emerge. Similarly, we can see that units learned for Europarl dataset focus to encode some key words (_e.g. vote_, _propose_, _environment_) in the training corpus.

To confirm that it also holds for representations learned in NLP tasks, we divide granularity of natural language concepts to the morpheme, word and \(N\)-gram phrase (\(N=2,3,4,5\)), and observe the number of units that they are aligned in different layers.

To validate this hypothesis, we re-train ByteNet from scratch while varying only layer depth of the encoder and fixing other conditions. We record their BLEU scores on the validation data as shown in Figure 6. The performance of the translation model does not change much with more than six encoder layers, but it significantly drops at the models with fewer than 4 encoder layers. This trend coincides with the result from Figure 5 that the evolution of concept granularity stops around middle-to-higher layers. This shared pattern suggests that about six encoder layers are enough to encode informative factors in the given datasets to perform optimally on the translation task. In deeper models, this may suggest that the middle layer's representation may be already informative enough to encode the input text, and our result may partly coincide with that of Mou et al. (2016), which shows that representation of intermediate layers is more transferable than that of deeper layers in language tasks, unlike in computer vision where deeper layers are usually more useful and discriminative.

We observe that the concepts do not appear uniformly; some concepts are aligned to many units, while others are aligned to few or even no units.

_Two possible hypotheses may explain the emergence of dominant concepts. First, the concepts with a higher frequency in training data may be aligned to more units.

Second, the concepts that have more influence on the objective function (expected loss) may be aligned to more units. We can measure the effect of concept \(c\) on the task performance as _Delta of Expected Loss_ (DEL) as follows: \[\mathsf{DEL}(c)=\mathbb{E}_{s\in\mathcal{S},y\in\mathcal{Y}}[\mathcal{L}(s,y) ]-\mathbb{E}_{s\in\mathcal{S},y\in\mathcal{Y}}[\mathcal{L}(\mathsf{Occ}_{c}(s ),y)] \tag{3}\]

As a result, \(\mathsf{DEL}(c)\) measures the impact of concept \(c\) on the loss function, where a large positive value implies that concept \(c\) has an important role for solving the target task.

We proposed a simple but highly effective concept alignment method for character-level CNNs to confirm that each unit of the hidden layers serves as detectors of natural language concepts. Using this method, we analyzed the characteristics of units with multiple datasets on classification and translation tasks.

Another direction is to quantify the properties of individual units in other models widely used in NLP tasks.

In section 3.2, we define Degree of Alignment (DoA) between concept \(c_{n}\) and unit \(u\) as activation value of unit \(u\) for replication of \(c_{n}\).

We then define DoA between candidate concept \(c_{n}\) and unit \(u\) by using PMI as follow: \[\text{DoA}_{u,c_{n}}=\text{pmi}(u,c_{n})=\log\frac{p(u,c_{n})}{p(u)p(c_{n})}, \text{where} \tag{5a}\] \[p(u)=\frac{\#[s\in\mathcal{S},s\in\text{topK}(u)]}{\#[s\in \mathcal{S}]},\] (5b) \[p(c_{n})=\frac{\#[s\in\mathcal{S},c_{n}\in s]}{\#[s\in \mathcal{S}]},\] (5c) \[p(u,c_{n})=\frac{\#[s\in\text{topK}(u),c_{n}\in s]}{\#[s\in \text{topK}(u)]} \tag{5d}\]

We tested concept alignments with the following concept occlusion method. For each of the top \(K\) sentences, we replace a \(c_{n}\) by dummy character tokens which have no meaning, forward it to the model, and measure the reduction of the unit activation value. We repeat this for every candidate concept in the sentences - as a result, we can identify which candidate concept greatly reduce unit activation values. We thus define concepts aligned to each unit as the candidate concept that consistently lower the unit activation across the top \(K\) sentences. More formally, for each unit \(u\), let \(\mathcal{S}=\{s_{1},...,s_{K}\}\) be top \(K\) activated sentences. Since we occlude each candidate concept in sentences, we define the set of candidate concept \(\mathcal{C}=\{c_{1},...,c_{N}\}\), obtained from parsing each sentence in \(\mathcal{S}\). We define the **degree of alignment (DoA)** between a concept \(c\in\mathcal{C}\) and a unit \(u\) as: \[\text{DoA}_{u,c_{n}}=\frac{1}{Z}\sum_{s\in\mathcal{S}}\mathbb{1}(c_{n}\in s)( a_{u}(s)-a_{u}(\text{Occ}_{c_{n}}(s))) \tag{6}\] where \(Z\) is a normalizing factor, and \(a_{u}\) indicates the mean activation of unit \(u\), \(\text{Occ}_{c_{n}}(s)\) is a sentence \(s\) where candidate concept \(c_{n}\) is occluded, and \(\mathbb{1}(c_{n}\in s)\) is an indicator of whether \(c_{n}\) is included in the sentence \(s\). In short, the DoA measures how much a candidate concept contributes to the activation of the unit's top \(K\) sentences. If a candidate concept \(c_{n}\) appears in the topsentences \(\mathcal{S}\) and greatly reduces the activation of unit \(u\), then \(\mathsf{DoA}_{u,c_{n}}\) gets large, implying that the \(c_{n}\) is strongly aligned to unit \(u\).

In Section 3.2, we set \(M=3\). Although \(M\) is used as a threshold to set how many concepts per unit are considered, different \(M\) values have little influence on quantitative results such as selectivity in Section 4.2. Figure 8 shows the mean and variance of selectivity values with different \(M=[1,3,5,10]\), where there is little variants in the overall trend; the sensitivity of the _replicate_ set is the highest, and that of _one instance_ is runner-up, and that of _random_ is the lowest.

We first define the unit interpretability for unit \(u\) as follows: \[\mathsf{Interpretability}(u)=\begin{cases}1&\text{if max}_{s\in\mathcal{S}}\{a _{u}(s)\}<\text{max}_{i=1,\dots,N}\{r_{i}\}\\ 0&\text{otherwise}\end{cases} \tag{7}\] where \(\mathcal{S}\) is the set of training sentences, \(a_{u}(s)\) is the activation value of unit \(u\), and \(r_{i}\) is the activation value of the sentence that is made up of replicating concept \(c_{i}\). We define unit \(u\) as interpretable when its \(\mathsf{Interpretability}(u)\) equals to 1, and otherwise as non-interpretable. The intuition is that if a replicated sentence that is composed of only one concept has a less activation value than the top-activated sentences, the unit is not sensitive to the concept compared to a sequence of different words.

We introduce some units whose concepts have the shared meaning in Section 4.3. We here refer _concept cluster_ to the concepts that are aligned to the same unit and have similar semantics or grammatical roles. We analyze how clusters are formed in the units and how they vary with the target task and layer depth. Each central heat map represents the number of times each concept pair is aligned to the same unit. Since the concepts in the x, y-axes are ordered by the clustering result, if the diagonal blocks (concept clusters) emerge more strongly, the concepts in the same unit are more likely to have the similar meanings.

We analyze how concept clusters change by layer in each task. We compute the averaged pairwise distance between the concepts in each layer. We project each concept to the vector space using the three pretrained embeddings: (1) Glove (Pennington et al., 2014), (2) ConceptNet (Speer et al., 2017), (3) fastText.

Consistent with our intuition, in all datasets, both the document frequency and the delta of expected loss are closely related to the number of units per concept. It concludes that the representations are learned for identifying not only the frequent concepts in the training set and but also the important concepts for solving the target task.

In the classification tasks, we expect to find more concepts that are directly related to predicting the output label, as opposed to the translation tasks where the representations may have to include information on most of the words for an accurate translation. While our goal is not to relate each concept to one of the labels, we find several concepts that are more predictive to a particular label than others. Consistent with section 4.4, there are data-specific and task-specific concepts aligned in each layer; _i.e._ {_worst, 2 stars, awful_} at Yelp Review, {_film, ship, school_} at DBpedia, and some key words at translation datasets.

Input: Figure 8: Mean and variance of selectivity values for different M = [1, 3, 5, 10], where M is the number of selected concepts per unit. In all settings, the selectivity of replicate ones is the highest, that of one instance ones is runner-up, and that of random is the lowest near 0.

Input: Figure 6: BLEU scores on the validation data for three translation models. We train ByteNet from scratch on each dataset by varying the number of encoder layers.

Input: Figure 4: 30 concepts selected by the number of aligned units in three encoding layers of ByteNet learned on the Europarl translation dataset (left column) and VDCNN learned on AG-News (right column). [#] symbol denotes morpheme concepts. See section 4.4 for details.

Input: Figure 3: Examples of top activated sentences and aligned concepts to some units in several encoding layers of ByteNet and VDCNN. For each unit, concepts and their presence in top K sentences are shown in the same color. [#] symbol denotes morpheme concepts. See section 4.3 for details.

Input: Figure 10: Examples of non-interpretable units, their concepts and top 5 activated sentences. Units are from representations learned on english-to-french translation dataset. See section D for details.

Input: Figure 10: Examples of non-interpretable units, their concepts and top 5 activated sentences. Units are from representations learned on english-to-french translation dataset. See section D for details.

Input: Figure 1: We discover the most activated sentences and aligned concepts to the units in hidden representations of deep convolutional networks. Aligned concepts appear frequently in most activated sentences, implying that those units respond selectively to specific natural language concepts.

Input: Figure 14: The Pearson correlations between the number of units per concept and (i) document frequency (top row), (ii) delta of expected loss (bottom row). They are measured at the final layer representation.

Input: Figure 15: 30 concepts selected by the number of aligned units in four encoding layers in VDCNN learned on Yelp Review dataset and DBpedia ontology dataset, and ByteNet learned on the English-to-German, English-to-French, and English-to-Czech parallel corpus. [#] symbol denotes morpheme concept.

Input: Figure 12: Averaged pairwise distances of concepts in each layer per task. Top and bottom row show the concept distances of translation models and classification models, respectively. For projecting concepts into the embedding space, we use three pretrained embedding: Glove, ConceptNet, FastText. See Appendix E.2 for more details.

Input: Figure 13: The number of unique concepts in each layer. It increases with the layer depth, which implies that the units in a deeper layer represent more diverse concepts.

Input: Figure 16: Aligned concepts per each task and their number of occurrences over multiple layers. See Appendix H for more details.

Input: Figure 11: Concept clusters of the last layer representations learned in each task. The more distinct the diagonal blocks are, the stronger the tendency that concepts aligned to the same unit share similar meanings or semantics. See Appendix E for details.

Input: Figure 14: The Pearson correlations between the number of units per concept and (i) document frequency (top row), (ii) delta of expected loss (bottom row). They are measured at the final layer representation.

Input: Figure 7: Correlations between the number of units per concept and (a) document frequency and (b) delta of expected loss. Pearson correlation coefficients are measured at the final layer of the AG News sentence classification task. Some concepts aligned to many units are annotated. Results on other tasks are available in Appendix F.

Input: Figure 15: 30 concepts selected by the number of aligned units in four encoding layers in VDCNN learned on Yelp Review dataset and DBpedia ontology dataset, and ByteNet learned on the English-to-German, English-to-French, and English-to-Czech parallel corpus. [#] symbol denotes morpheme concept.

Input: Figure 16: Aligned concepts per each task and their number of occurrences over multiple layers. See Appendix H for more details.

Input: Figure 2: Mean and variance of selectivity values over all units in the learned representation for each dataset. Sentences including the concepts that our alignment method discovers always activate units significantly more than random sentences. See section 4.2 for details.

Input: Figure 11: Concept clusters of the last layer representations learned in each task. The more distinct the diagonal blocks are, the stronger the tendency that concepts aligned to the same unit share similar meanings or semantics. See Appendix E for details.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence, the claim about proposing a concept alignment method appears mostly well-stated, as the paper provides:

1. A detailed mathematical formulation of their method using Degree of Alignment (DoA) metrics and selectivity measures
2. Comprehensive empirical validation across multiple datasets and model architectures
3. Clear explanations of how concepts are identified and aligned to units
4. Quantitative evaluation metrics to assess the effectiveness
5. Analysis of limitations and alternative approaches

The reviewer's comment also supports the claim's validity by noting the paper is "well written and structured" with "interesting and novel findings." While some limitations exist (e.g., lack of empirical study on concept coverage), the core claim about proposing the method is appropriately scoped and supported by substantial evidence.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In order to quantitatively analyze such an intriguing phenomenon, we propose a concept alignment method based on how units respond to the replicated text.

The review comment to be evaluated is:

========== Edit following authors' response  ==========

Thank you for your detailed response and updated version. I think the new revision is significantly improved, mainly in more quantitative analyses and details in several places. I have updated my evaluation accordingly.

See a few more points below.

1. Thank you for clarifying your definition of concepts. I still think that the word "concept" has a strong semantic connotation, while the linguistic elements your analyses capture may do other things. The results in appendix E do show that some semantic clusters arise. It's especially interesting to see the blocks in some of the heat maps, where similar "concepts" are clustered together (like the sports terms in AG); consider commenting on this.

2. The new quantitative analyses are helpful. One other suggestion that I mentioned before is to connect detected concepts to external resources like WordNet or ConceptNet. That would help show that "concepts" are indeed semantic objects.

3. The motivation for replicating as normalizing for length does make sense, although the input would still be unnatural. The comparison to "one instance" is helpful, but it's interesting that the differences between it and replication in figure 2 are not large. It would be good to show results that substantiate your assumption that without replication there will be a bias towards lengthy concepts. Does "one instance" detect more lengthy concepts than replication?

4. The results on frequency and loss difference in 4.5 are very interesting. There is another angle to consider frequency: words that appear frequently often carry less semantic content (e.g. function words), so one might conjecture that they would require less units. It may be interesting to look at which concepts are detected at each frequency bin.

5. Minor points: section 2.2 still mentions "regression" where it should be "classification".

6. A few remaining grammar issues:
- "one concept has a less activation value.." - rephrase
- end of section 3.3: "this experiments" -> "these experiments"

========== Original review follows ==========

Summary:
=======
This paper analyzes individual units in CNN models for text classification and translation tasks. It defines a measure of sensitivity for a unit and evaluates how sensitive each unit is to "concepts" in the input text, where concepts are morphemes, words, and phrases. The analysis shows that some units seem to learn semantic concepts, while others capture linguistic elements that are frequent or relevant for the end task. Layer-wise results show some correspondence between layer depth and linguistic element size.

The paper studies an important question that is relatively under-studies in NLP compared to the computer vision community. The motivation for the work is quite convincing.
I found some of the results and analysis interesting, but overall felt that the work can be made much stronger by more quantitative evaluations. I am also worried that the notion of "concept" is misleading here. See below for this and other comments. I am willing to reconsider my evaluation pending response to the below issues.

Main comments:
=============
1. Concepts:
- morphemes, words, and phrases - are these "concepts"? They are indeed "fundamental building blocks of natural language" (2.2), but "concepts" has a more semantic connotation that I'm not sure these units target at.
- Some of the results do suggest that units learn concepts, as the analysis in 4.2 shows a "unit detecting the meaning of certainty in knowledge" and later units that have similar sentiments. It would be informative to quantify this in some way, for example by matching detected concepts to WordNet synsets, sentiment lexicons, etc., or else tagging and classifying them with various NLP tools. This could also reveal if units learn more syntactic or semantic concepts, and so on.
2. Generally, many of the analyses in the paper are qualitative and on a small scale. The results will be more convincing with more automatic aggregate measures.
3. The structure of the paper is confusing. Section3 starts with the approach but then mentions datasets and tasks (3.1). Section 4 is titled experiments, but section 4.1 starts with defining the concept selectivity. I would suggest reorganizing sections 3 and 4, such that section 3 describes all the methods and metrics, while dataset-specific parts are moved to section 4.
4. section 3.2 should provide more details on the sentence representation and how its obtained in the CNN models. A mathematical derivation and/or figure could be helpful. It is also not clear to me what's the motivation for mean-pooling over the l entries of the vector.
5. section 3.3: the use of replicated text for "concept alignment" is puzzling. This is not a natural input to the model, and I think more justification and motivation åre needed for this issue, as well as perhaps comparison with other approaches.
6. I found section 4.4 very interesting. It shows some intuitive results of larger linguistic elements learned at higher layers, but then some results that do not show such a trend. Then, hypothesizing that the middle layers are sufficient AND validating the hypothesis by retraining the model is excellent. It's a very nice demonstration that the analysis can lead to model improvements.
7. Figure 2 seems to be almost caused by construction of the different options for S_+. Is it surprising that the replicate set has the highest sensitivity? Is there a better control setup than comparing with a random set?
8. One concern that I have is the effect of confounding factors like frequency on the results. The papers occasionally attributes importance to concepts (e.g. in 4.2), but I wonder if instead we may be seeing more frequent words. Controlling for the effect of frequency would be useful.

Minor comments:
==============
- Section 2.2, first paragraph: regression should be changed to classification
- The related work is generally relevant, although one could mention a few other papers that analyzed individual neurons in NLP tasks [1, 2]
- section 4.1: the random set may perhaps be denoted by something more neutral, not S_+ as the replicate and inclusion sets.
- section 4.3, last paragraph: listing examples showing that units in Europarl focus on key words would be good.
- Figure 5, y axis label: should this be number of units instead of concepts?
- Appendix A has several interesting points but there is no reference to them from the main paper.

Writing, grammar, etc.:
======================
- Introduction: among them - who is them?
- 2.1: motivated from -> motivated by; In computer vision community -> In the computer vision community
- 2.1: quantifying characteristics of representations in layer-wise -> rephrase
- 3.2: dimension of sentence -> dimension of the/a sentence
- 4.1: to which -> remove "which"
- 4.2: in the several encoding layer -> in several encoding layers
- 4.3: aliged -> aligned
- Capitalize titles in references
- A.2: with following -> with the following; how much candidate -> how much a candidate; consider following -> consider the following
- A.3: induces similar bias -> induces a bias; such phrase -> such a phrase; on very -> on a very
- C: where model -> where the model; In consistent -> Consistent; where model -> where the model

References
==========
[1] Qian et al., Analyzing linguistic knowledge in sequential model of sentence
[2] Shi et al., Why Neural Translations are the Right Length

The evidence to be evaluated is:Compared with these previous studies, we focus on discovering a much wider variety of _natural language concepts_, including any morphemes, words, and phrases all found in the training data. To the best our knowledge, this is the first attempt to discover concepts among all that exist in the form of natural language from the training corpus. By extending the scope of detected concepts to meaningful building blocks of natural language, we provide insights into how various linguistic features are encoded by the hidden units of deep representations.

(1) By identifying the role of the individual units, rather than analyzing the representation as a whole, we provide more fine-grained understanding of how the representations encode informative factors in training data.

(3) Our concept alignment method does not need any additional labeled data or re-training process, so it can always provide deterministic interpretation results using only the training data.

Our approach for aligning natural language concepts to units is summarized as follows. We first train a CNN model for each natural language task (_e.g._ translation and classification) and retrieve training sentences that highly activate specific units. Interestingly, we discover morphemes, words and phrases that appear dominantly within these retrieved sentences, implying that those concepts have a significant impact on the activation value of the unit. Then, we find a set of concepts which attribute a lot to the unit activation by measuring activation value of each replicated candidate concept, and align them to unit.

Given a layer and sentence \(s\in\mathcal{S}\), let \(\tilde{A}^{t}_{u}(s)\)denote the activation of unit \(u\) at spatial location \(l\). Then, for unit \(u\), we average activations over all spatial locations as \(a_{u}(s)=\frac{1}{Z}\sum_{l}A_{u}^{l}(s)\), where \(Z\) is a normalizer. We then retrieve top \(K\) training sentences per unit with the highest mean activation \(a_{u}\).

We propose a simple approach for identifying the concepts as follows. For constructing candidate concepts, we parse each of top \(K\) sentences with a constituency parser (Kitaev and Klein, 2018). Within the constituency-based parse tree, we define candidate concepts as all terminal and non-terminal nodes (_e.g._ from sentence _John hit the balls_, we obtain candidate concepts as {_John, hit, the balls, hit the balls, John hit the balls_}). We also break each word into morphemes using a morphological analysis tool (Virpioja et al., 2013) and add them to candidate concepts (_e.g._ from word _balls_, we obtain morphemes {_ball, s_}). We repeat this process for every top \(K\) sentence and build a set of candidate concepts for unit \(u\), which is denoted as \(\mathcal{C}_{u}=\{c_{1},...,c_{N}\}\), where \(N\) is the number of candidate concepts of the unit. Next, we measure how each candidate concept contributes to the unit's activation value. For normalizing the degree of an input signal to the unit activation, we create a synthetic sentence by replicating each candidate concept so that its length is identical to the average length of all training sentences (_e.g._ candidate concept _the ball_ is replicated as _the ball the ball the ball_...). Replicated sentences are denoted as \(\mathcal{R}=\{r_{1},...,r_{N}\}\), and each \(r_{n}\in\mathcal{R}\) is forwarded to CNN, and their activation value of unit \(u\) is measured as \(a_{u}(r_{n})\in\mathbb{R}\), which is averaged over \(l\) entries. Finally, the degree of alignment (DoA) between a candidate concept \(c_{n}\) and a unit \(u\) is defined as follows: \[\mathsf{DoA}_{u,c_{n}}=a_{u}(r_{n}) \tag{1}\] In short, the \(\mathsf{DoA}\)1 measures the extent to unit \(u\)'s activation is sensitive to the presence of candidate concept \(c_{n}\). If a candidate concept \(c_{n}\) appears in the top \(K\) sentences and unit's activation value \(a_{u}\) is responsive to \(c_{n}\) a lot, then \(\mathsf{DoA}_{u,c_{n}}\) gets large, suggesting that candidate concept \(c_{n}\) is strongly aligned to unit \(u\).

Finally, for each unit \(u\), we define a set of its aligned concepts \(\mathcal{C}_{u}^{*}=\{c_{1}^{*},...,c_{M}^{*}\}\) as \(M\) candidate concepts with the largest DoA values in \(C_{u}\). Depending on how we set \(M\), we can detect different numbers of concepts per unit.

To quantitatively evaluate how well our approach aligns concepts, we measure how selectively each unit responds to the aligned concept. Motivated by Morcos et al. (2018), we define the **concept selectivity** of a unit \(u\), to a set of concepts \(\mathcal{C}_{u}^{*}\) that our alignment method detects, as follows: \[\mathsf{Sel}_{u}=\frac{\mu_{+}-\mu_{-}}{\max_{s\in\mathcal{S}}a_{u}(s)-\min_{s \in\mathcal{S}}a_{u}(s)} \tag{2}\] where \(\mathcal{S}\) denotes all sentences in training set, and \(\mu_{+}=\frac{1}{|\mathcal{S}_{+}|}\sum_{s\in\mathcal{S}_{+}}a_{u}(s)\) is the average value of unit activation when forwarding a set of sentences \(\mathcal{S}_{+}\), which is defined as one of the following: * _replicate_: \(\mathcal{S}_{+}\) contains the sentences created by replicating each concept in \(\mathcal{C}_{u}^{*}\). As before, the sentence length is set as the average length of all training sentences for fair comparison.

In contrast, \(\mu_{-}=\frac{1}{|\mathcal{S}_{-}|}\sum_{s\in\mathcal{S}_{-}}a_{u}(s)\) is the average value of unit activation when forwarding \(\mathcal{S}_{-}\), which consists of training sentences that do _not_ include any concept in \(\mathcal{C}_{u}^{*}\). Intuitively, if unit \(u\)'s activation is highly sensitive to \(\mathcal{C}_{u}^{*}\) (_i.e._ those found by our alignment method) and if it is not to other factors, then \(\mathsf{Sel}_{u}\) gets large; otherwise, \(\mathsf{Sel}_{u}\) is near 0. More interestingly, we discover that many units could capture specific meanings or syntactic roles beyond superficial, low-level patterns. For example, unit 690 of the 14th layer in ByteNet captures (_what, who, where_) concepts, all of which play a similar grammatical role. On the other hand, unit 224 of the 14th layer in ByteNet and unit 53 of the 0th layer in VDCNN each captures semantically similar concepts, with the ByteNet unit detecting the meaning of certainty in knowledge (_sure, know, aware_) and the VDCNN unit detecting years (_1999, 1969, 1992_).

We note that there are units that detect concepts more abstract than just morphemes, words, or phrases, and for these units, our method tends to align relevant lower-level concepts. For example, in unit 244 of the 3rd layer in VDCNN, while each aligned concept emerges only once in the top \(K\) sentences, all top \(K\) sentences have similar _nucances_ like positive sentiments. In this case, our method does capture relevant phrase-level concepts (e.g., _very disappointing, absolute worst place_), indicating that the higher-level _nucance_ (e.g., negativity) is indirectly captured.

Our approach thus tends to find concepts that are frequent in training data or considered as more important than others for solving the target task.

* Several units in the CNN learned on NLP tasks respond selectively to specific natural language concepts, rather than getting activated in an uninterpretable way. This means that these units can serve as detectors for specific natural language concepts.

* There are units capturing syntactically or semantically related concepts, suggesting that they model the _meaning or grammatical role shared between those concepts_, as opposed to superficially modeling each natural language _symbol_.

Overall, we find that data and task-specific concepts are likely to be aligned to many units. In AG News, since the task is to classify given sentences into following categories; _World_, _Sports_, _Business_ and _Science/Tech_, concepts related to these topics commonly emerge. Similarly, we can see that units learned for Europarl dataset focus to encode some key words (_e.g. vote_, _propose_, _environment_) in the training corpus.

To confirm that it also holds for representations learned in NLP tasks, we divide granularity of natural language concepts to the morpheme, word and \(N\)-gram phrase (\(N=2,3,4,5\)), and observe the number of units that they are aligned in different layers.

To validate this hypothesis, we re-train ByteNet from scratch while varying only layer depth of the encoder and fixing other conditions. We record their BLEU scores on the validation data as shown in Figure 6. The performance of the translation model does not change much with more than six encoder layers, but it significantly drops at the models with fewer than 4 encoder layers. This trend coincides with the result from Figure 5 that the evolution of concept granularity stops around middle-to-higher layers. This shared pattern suggests that about six encoder layers are enough to encode informative factors in the given datasets to perform optimally on the translation task. In deeper models, this may suggest that the middle layer's representation may be already informative enough to encode the input text, and our result may partly coincide with that of Mou et al. (2016), which shows that representation of intermediate layers is more transferable than that of deeper layers in language tasks, unlike in computer vision where deeper layers are usually more useful and discriminative.

We observe that the concepts do not appear uniformly; some concepts are aligned to many units, while others are aligned to few or even no units.

_Two possible hypotheses may explain the emergence of dominant concepts. First, the concepts with a higher frequency in training data may be aligned to more units.

Second, the concepts that have more influence on the objective function (expected loss) may be aligned to more units. We can measure the effect of concept \(c\) on the task performance as _Delta of Expected Loss_ (DEL) as follows: \[\mathsf{DEL}(c)=\mathbb{E}_{s\in\mathcal{S},y\in\mathcal{Y}}[\mathcal{L}(s,y) ]-\mathbb{E}_{s\in\mathcal{S},y\in\mathcal{Y}}[\mathcal{L}(\mathsf{Occ}_{c}(s ),y)] \tag{3}\]

As a result, \(\mathsf{DEL}(c)\) measures the impact of concept \(c\) on the loss function, where a large positive value implies that concept \(c\) has an important role for solving the target task.

We proposed a simple but highly effective concept alignment method for character-level CNNs to confirm that each unit of the hidden layers serves as detectors of natural language concepts. Using this method, we analyzed the characteristics of units with multiple datasets on classification and translation tasks.

Another direction is to quantify the properties of individual units in other models widely used in NLP tasks.

In section 3.2, we define Degree of Alignment (DoA) between concept \(c_{n}\) and unit \(u\) as activation value of unit \(u\) for replication of \(c_{n}\).

We then define DoA between candidate concept \(c_{n}\) and unit \(u\) by using PMI as follow: \[\text{DoA}_{u,c_{n}}=\text{pmi}(u,c_{n})=\log\frac{p(u,c_{n})}{p(u)p(c_{n})}, \text{where} \tag{5a}\] \[p(u)=\frac{\#[s\in\mathcal{S},s\in\text{topK}(u)]}{\#[s\in \mathcal{S}]},\] (5b) \[p(c_{n})=\frac{\#[s\in\mathcal{S},c_{n}\in s]}{\#[s\in \mathcal{S}]},\] (5c) \[p(u,c_{n})=\frac{\#[s\in\text{topK}(u),c_{n}\in s]}{\#[s\in \text{topK}(u)]} \tag{5d}\]

We tested concept alignments with the following concept occlusion method. For each of the top \(K\) sentences, we replace a \(c_{n}\) by dummy character tokens which have no meaning, forward it to the model, and measure the reduction of the unit activation value. We repeat this for every candidate concept in the sentences - as a result, we can identify which candidate concept greatly reduce unit activation values. We thus define concepts aligned to each unit as the candidate concept that consistently lower the unit activation across the top \(K\) sentences. More formally, for each unit \(u\), let \(\mathcal{S}=\{s_{1},...,s_{K}\}\) be top \(K\) activated sentences. Since we occlude each candidate concept in sentences, we define the set of candidate concept \(\mathcal{C}=\{c_{1},...,c_{N}\}\), obtained from parsing each sentence in \(\mathcal{S}\). We define the **degree of alignment (DoA)** between a concept \(c\in\mathcal{C}\) and a unit \(u\) as: \[\text{DoA}_{u,c_{n}}=\frac{1}{Z}\sum_{s\in\mathcal{S}}\mathbb{1}(c_{n}\in s)( a_{u}(s)-a_{u}(\text{Occ}_{c_{n}}(s))) \tag{6}\] where \(Z\) is a normalizing factor, and \(a_{u}\) indicates the mean activation of unit \(u\), \(\text{Occ}_{c_{n}}(s)\) is a sentence \(s\) where candidate concept \(c_{n}\) is occluded, and \(\mathbb{1}(c_{n}\in s)\) is an indicator of whether \(c_{n}\) is included in the sentence \(s\). In short, the DoA measures how much a candidate concept contributes to the activation of the unit's top \(K\) sentences. If a candidate concept \(c_{n}\) appears in the topsentences \(\mathcal{S}\) and greatly reduces the activation of unit \(u\), then \(\mathsf{DoA}_{u,c_{n}}\) gets large, implying that the \(c_{n}\) is strongly aligned to unit \(u\).

In Section 3.2, we set \(M=3\). Although \(M\) is used as a threshold to set how many concepts per unit are considered, different \(M\) values have little influence on quantitative results such as selectivity in Section 4.2. Figure 8 shows the mean and variance of selectivity values with different \(M=[1,3,5,10]\), where there is little variants in the overall trend; the sensitivity of the _replicate_ set is the highest, and that of _one instance_ is runner-up, and that of _random_ is the lowest.

We first define the unit interpretability for unit \(u\) as follows: \[\mathsf{Interpretability}(u)=\begin{cases}1&\text{if max}_{s\in\mathcal{S}}\{a _{u}(s)\}<\text{max}_{i=1,\dots,N}\{r_{i}\}\\ 0&\text{otherwise}\end{cases} \tag{7}\] where \(\mathcal{S}\) is the set of training sentences, \(a_{u}(s)\) is the activation value of unit \(u\), and \(r_{i}\) is the activation value of the sentence that is made up of replicating concept \(c_{i}\). We define unit \(u\) as interpretable when its \(\mathsf{Interpretability}(u)\) equals to 1, and otherwise as non-interpretable. The intuition is that if a replicated sentence that is composed of only one concept has a less activation value than the top-activated sentences, the unit is not sensitive to the concept compared to a sequence of different words.

We introduce some units whose concepts have the shared meaning in Section 4.3. We here refer _concept cluster_ to the concepts that are aligned to the same unit and have similar semantics or grammatical roles. We analyze how clusters are formed in the units and how they vary with the target task and layer depth. Each central heat map represents the number of times each concept pair is aligned to the same unit. Since the concepts in the x, y-axes are ordered by the clustering result, if the diagonal blocks (concept clusters) emerge more strongly, the concepts in the same unit are more likely to have the similar meanings.

We analyze how concept clusters change by layer in each task. We compute the averaged pairwise distance between the concepts in each layer. We project each concept to the vector space using the three pretrained embeddings: (1) Glove (Pennington et al., 2014), (2) ConceptNet (Speer et al., 2017), (3) fastText.

Consistent with our intuition, in all datasets, both the document frequency and the delta of expected loss are closely related to the number of units per concept. It concludes that the representations are learned for identifying not only the frequent concepts in the training set and but also the important concepts for solving the target task.

In the classification tasks, we expect to find more concepts that are directly related to predicting the output label, as opposed to the translation tasks where the representations may have to include information on most of the words for an accurate translation. While our goal is not to relate each concept to one of the labels, we find several concepts that are more predictive to a particular label than others. Consistent with section 4.4, there are data-specific and task-specific concepts aligned in each layer; _i.e._ {_worst, 2 stars, awful_} at Yelp Review, {_film, ship, school_} at DBpedia, and some key words at translation datasets.

Input: Figure 8: Mean and variance of selectivity values for different M = [1, 3, 5, 10], where M is the number of selected concepts per unit. In all settings, the selectivity of replicate ones is the highest, that of one instance ones is runner-up, and that of random is the lowest near 0.

Input: Figure 6: BLEU scores on the validation data for three translation models. We train ByteNet from scratch on each dataset by varying the number of encoder layers.

Input: Figure 4: 30 concepts selected by the number of aligned units in three encoding layers of ByteNet learned on the Europarl translation dataset (left column) and VDCNN learned on AG-News (right column). [#] symbol denotes morpheme concepts. See section 4.4 for details.

Input: Figure 3: Examples of top activated sentences and aligned concepts to some units in several encoding layers of ByteNet and VDCNN. For each unit, concepts and their presence in top K sentences are shown in the same color. [#] symbol denotes morpheme concepts. See section 4.3 for details.

Input: Figure 10: Examples of non-interpretable units, their concepts and top 5 activated sentences. Units are from representations learned on english-to-french translation dataset. See section D for details.

Input: Figure 10: Examples of non-interpretable units, their concepts and top 5 activated sentences. Units are from representations learned on english-to-french translation dataset. See section D for details.

Input: Figure 1: We discover the most activated sentences and aligned concepts to the units in hidden representations of deep convolutional networks. Aligned concepts appear frequently in most activated sentences, implying that those units respond selectively to specific natural language concepts.

Input: Figure 14: The Pearson correlations between the number of units per concept and (i) document frequency (top row), (ii) delta of expected loss (bottom row). They are measured at the final layer representation.

Input: Figure 15: 30 concepts selected by the number of aligned units in four encoding layers in VDCNN learned on Yelp Review dataset and DBpedia ontology dataset, and ByteNet learned on the English-to-German, English-to-French, and English-to-Czech parallel corpus. [#] symbol denotes morpheme concept.

Input: Figure 12: Averaged pairwise distances of concepts in each layer per task. Top and bottom row show the concept distances of translation models and classification models, respectively. For projecting concepts into the embedding space, we use three pretrained embedding: Glove, ConceptNet, FastText. See Appendix E.2 for more details.

Input: Figure 13: The number of unique concepts in each layer. It increases with the layer depth, which implies that the units in a deeper layer represent more diverse concepts.

Input: Figure 16: Aligned concepts per each task and their number of occurrences over multiple layers. See Appendix H for more details.

Input: Figure 11: Concept clusters of the last layer representations learned in each task. The more distinct the diagonal blocks are, the stronger the tendency that concepts aligned to the same unit share similar meanings or semantics. See Appendix E for details.

Input: Figure 14: The Pearson correlations between the number of units per concept and (i) document frequency (top row), (ii) delta of expected loss (bottom row). They are measured at the final layer representation.

Input: Figure 7: Correlations between the number of units per concept and (a) document frequency and (b) delta of expected loss. Pearson correlation coefficients are measured at the final layer of the AG News sentence classification task. Some concepts aligned to many units are annotated. Results on other tasks are available in Appendix F.

Input: Figure 15: 30 concepts selected by the number of aligned units in four encoding layers in VDCNN learned on Yelp Review dataset and DBpedia ontology dataset, and ByteNet learned on the English-to-German, English-to-French, and English-to-Czech parallel corpus. [#] symbol denotes morpheme concept.

Input: Figure 16: Aligned concepts per each task and their number of occurrences over multiple layers. See Appendix H for more details.

Input: Figure 2: Mean and variance of selectivity values over all units in the learned representation for each dataset. Sentences including the concepts that our alignment method discovers always activate units significantly more than random sentences. See section 4.2 for details.

Input: Figure 11: Concept clusters of the last layer representations learned in each task. The more distinct the diagonal blocks are, the stronger the tendency that concepts aligned to the same unit share similar meanings or semantics. See Appendix E for details.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, I find this claim to be well-stated and accurately reflects the methodology presented. The evidence shows a detailed mathematical framework for concept alignment based on unit responses to replicated text, including clear formulas for Degree of Alignment (DoA) and selectivity measurements. The method is thoroughly validated through multiple experiments and visualizations demonstrating how units respond to replicated concepts. The review comments also acknowledge the improved quantitative analyses while suggesting some refinements. The claim conservatively describes the basic approach without overstating its capabilities.

<score>1</score>

Justification:
1. The evidence provides comprehensive mathematical formulation (equations 1-2) that directly supports the claim about using unit responses to replicated text
2. Multiple figures (2, 8) demonstrate the effectiveness of the replication-based method compared to alternatives
3. The reviewer acknowledges the improved quantitative analyses while suggesting reasonable extensions rather than identifying major flaws
4. The claim is modestly worded, focusing on the basic methodological approach without overstating its implications or capabilities