Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_What natural language concepts are captured by each unit in the representations learned from NLP tasks?

The evidence to be evaluated is:Recent works on interpreting hidden representations at unit-level were mostly motivated by their counterparts in computer vision.

In the NLP domain, studies including Karpathy et al. (2015); Tang et al. (2017); Qian et al. (2016); Shi et al. (2016) analyzed the internal mechanisms of deep models used for NLP and found intriguing properties that appear in units of hidden representations.

Compared with these previous studies, we focus on discovering a much wider variety of _natural language concepts_, including any morphemes, words, and phrases all found in the training data. To the best our knowledge, this is the first attempt to discover concepts among all that exist in the form of natural language from the training corpus. By extending the scope of detected concepts to meaningful building blocks of natural language, we provide insights into how various linguistic features are encoded by the hidden units of deep representations. Most previous work that analyzes the learned representation of NLP tasks focused on constructing downstream tasks that predict concepts of interest. A common approach is to measure the performance of a classification model that predicts the concept of interest to see whether those concepts are encoded in representation of a input sentence. For example, Conneau et al. (2018); Adi et al. (2017); Zhu et al. (2018) proposed several probing tasks to test whether the (non-)linear regression model can predict well the syntactic or semantic information from the representation learned on translation tasks or the skip-thought or word embedding vectors. Shi et al. (2016); Belinkov et al. (2017) constructed regression tasks that predict labels such as voice, tense, part-of-speech tag, and morpheme from the encoder representation of the learned model in translation task.

(1) By identifying the role of the individual units, rather than analyzing the representation as a whole, we provide more fine-grained understanding of how the representations encode informative factors in training data. (2) Rather than limiting the linguistic features within the representation to be discovered, we focus on covering concepts of fundamental building blocks of natural language (morphemes, words, and phrases) present in the training data, providing more flexible interpretation results without relying on a predefined set of concepts.

We first train a CNN model for each natural language task (_e.g._ translation and classification) and retrieve training sentences that highly activate specific units. Interestingly, we discover morphemes, words and phrases that appear dominantly within these retrieved sentences, implying that those concepts have a significant impact on the activation value of the unit. Then, we find a set of concepts which attribute a lot to the unit activation by measuring activation value of each replicated candidate concept, and align them to unit.

Interestingly, some natural language patterns such as morphemes, words, phrases frequently appear in the retrieved sentences (see Figure 1), implying that those concepts might have a large attribution to the activation value of that unit.

For constructing candidate concepts, we parse each of top \(K\) sentences with a constituency parser (Kitaev and Klein, 2018). Within the constituency-based parse tree, we define candidate concepts as all terminal and non-terminal nodes (_e.g._ from sentence _John hit the balls_, we obtain candidate concepts as {_John, hit, the balls, hit the balls, John hit the balls_}). We also break each word into morphemes using a morphological analysis tool (Virpioja et al., 2013) and add them to candidate concepts (_e.g._ from word _balls_, we obtain morphemes {_ball, s_}). We repeat this process for every top \(K\) sentence and build a set of candidate concepts for unit \(u\), which is denoted as \(\mathcal{C}_{u}=\{c_{1},...,c_{N}\}\), where \(N\) is the number of candidate concepts of the unit.

Finally, the degree of alignment (DoA) between a candidate concept \(c_{n}\) and a unit \(u\) is defined as follows: \[\mathsf{DoA}_{u,c_{n}}=a_{u}(r_{n}) \tag{1}\] In short, the \(\mathsf{DoA}\)1 measures the extent to unit \(u\)'s activation is sensitive to the presence of candidate concept \(c_{n}\). If a candidate concept \(c_{n}\) appears in the top \(K\) sentences and unit's activation value \(a_{u}\) is responsive to \(c_{n}\) a lot, then \(\mathsf{DoA}_{u,c_{n}}\) gets large, suggesting that candidate concept \(c_{n}\) is strongly aligned to unit \(u\).

Finally, for each unit \(u\), we define a set of its aligned concepts \(\mathcal{C}_{u}^{*}=\{c_{1}^{*},...,c_{M}^{*}\}\) as \(M\) candidate concepts with the largest DoA values in \(C_{u}\). Depending on how we set \(M\), we can detect different numbers of concepts per unit.

We then focus on the representations in each encoder layer of ByteNet and convolutional layer of VDCNN, because as Mou et al. (2016) pointed out, the representation of the decoder (the output layer in the case of classification) is specialized for predicting the output of the target task rather than for learning the semantics of the input text. To quantitatively evaluate how well our approach aligns concepts, we measure how selectively each unit responds to the aligned concept. Motivated by Morcos et al. (2018), we define the **concept selectivity** of a unit \(u\), to a set of concepts \(\mathcal{C}_{u}^{*}\) that our alignment method detects, as follows: \[\mathsf{Sel}_{u}=\frac{\mu_{+}-\mu_{-}}{\max_{s\in\mathcal{S}}a_{u}(s)-\min_{s \in\mathcal{S}}a_{u}(s)} \tag{2}\] where \(\mathcal{S}\) denotes all sentences in training set, and \(\mu_{+}=\frac{1}{|\mathcal{S}_{+}|}\sum_{s\in\mathcal{S}_{+}}a_{u}(s)\) is the average value of unit activation when forwarding a set of sentences \(\mathcal{S}_{+}\), which is defined as one of the following: * _replicate_: \(\mathcal{S}_{+}\) contains the sentences created by replicating each concept in \(\mathcal{C}_{u}^{*}\). As before, the sentence length is set as the average length of all training sentences for fair comparison.

More interestingly, we discover that many units could capture specific meanings or syntactic roles beyond superficial, low-level patterns. For example, unit 690 of the 14th layer in ByteNet captures (_what, who, where_) concepts, all of which play a similar grammatical role. On the other hand, unit 224 of the 14th layer in ByteNet and unit 53 of the 0th layer in VDCNN each captures semantically similar concepts, with the ByteNet unit detecting the meaning of certainty in knowledge (_sure, know, aware_) and the VDCNN unit detecting years (_1999, 1969, 1992_). This suggests that, although we train character-level CNNs with feeding sentences as the form of discrete symbols (_i.e._ character indices), individual units could capture natural language concepts sharing a similar semantic or grammatical role.

We note that there are units that detect concepts more abstract than just morphemes, words, or phrases, and for these units, our method tends to align relevant lower-level concepts. For example, in unit 244 of the 3rd layer in VDCNN, while each aligned concept emerges only once in the top \(K\) sentences, all top \(K\) sentences have similar _nucances_ like positive sentiments. In this case, our method does capture relevant phrase-level concepts (e.g., _very disappointing, absolute worst place_), indicating that the higher-level _nucance_ (e.g., negativity) is indirectly captured.

* Several units in the CNN learned on NLP tasks respond selectively to specific natural language concepts, rather than getting activated in an uninterpretable way. This means that these units can serve as detectors for specific natural language concepts.

* There are units capturing syntactically or semantically related concepts, suggesting that they model the _meaning or grammatical role shared between those concepts_, as opposed to superficially modeling each natural language _symbol_.

Overall, we find that data and task-specific concepts are likely to be aligned to many units. In AG News, since the task is to classify given sentences into following categories; _World_, _Sports_, _Business_ and _Science/Tech_, concepts related to these topics commonly emerge. Similarly, we can see that units learned for Europarl dataset focus to encode some key words (_e.g. vote_, _propose_, _environment_) in the training corpus.

To confirm that it also holds for representations learned in NLP tasks, we divide granularity of natural language concepts to the morpheme, word and \(N\)-gram phrase (\(N=2,3,4,5\)), and observe the number of units that they are aligned in different layers.

In deeper models, this may suggest that the middle layer's representation may be already informative enough to encode the input text, and our result may partly coincide with that of Mou et al. (2016), which shows that representation of intermediate layers is more transferable than that of deeper layers in language tasks, unlike in computer vision where deeper layers are usually more useful and discriminative.

We observe that the concepts do not appear uniformly; some concepts are aligned to many units, while others are aligned to few or even no units.

First, the concepts with a higher frequency in training data may be aligned to more units. Figure 7-(a) shows the correlation between the frequency of each concept in the training corpus and the number of units where each concept is aligned in the last layer of the topic classification model learned on AG News dataset. Second, the concepts that have more influence on the objective function (expected loss) may be aligned to more units.

Another direction is to quantify the properties of individual units in other models widely used in NLP tasks. In particular, combining our definition of concepts with the attention mechanism (_e.g._ Bahdanau et al. (2015)) could be a promising direction, because it can reveal how the representations are attended by the model to capture concepts, helping us better understand the decision-making process of popular deep models.

In section 3.2, we define Degree of Alignment (DoA) between concept \(c_{n}\) and unit \(u\) as activation value of unit \(u\) for replication of \(c_{n}\).

We then define DoA between candidate concept \(c_{n}\) and unit \(u\) by using PMI as follow: \[\text{DoA}_{u,c_{n}}=\text{pmi}(u,c_{n})=\log\frac{p(u,c_{n})}{p(u)p(c_{n})}, \text{where} \tag{5a}\] \[p(u)=\frac{\#[s\in\mathcal{S},s\in\text{topK}(u)]}{\#[s\in \mathcal{S}]},\] (5b) \[p(c_{n})=\frac{\#[s\in\mathcal{S},c_{n}\in s]}{\#[s\in \mathcal{S}]},\] (5c) \[p(u,c_{n})=\frac{\#[s\in\text{topK}(u),c_{n}\in s]}{\#[s\in \text{topK}(u)]} \tag{5d}\]

We tested concept alignments with the following concept occlusion method. For each of the top \(K\) sentences, we replace a \(c_{n}\) by dummy character tokens which have no meaning, forward it to the model, and measure the reduction of the unit activation value. We repeat this for every candidate concept in the sentences - as a result, we can identify which candidate concept greatly reduce unit activation values. We thus define concepts aligned to each unit as the candidate concept that consistently lower the unit activation across the top \(K\) sentences. More formally, for each unit \(u\), let \(\mathcal{S}=\{s_{1},...,s_{K}\}\) be top \(K\) activated sentences. Since we occlude each candidate concept in sentences, we define the set of candidate concept \(\mathcal{C}=\{c_{1},...,c_{N}\}\), obtained from parsing each sentence in \(\mathcal{S}\). We define the **degree of alignment (DoA)** between a concept \(c\in\mathcal{C}\) and a unit \(u\) as: \[\text{DoA}_{u,c_{n}}=\frac{1}{Z}\sum_{s\in\mathcal{S}}\mathbb{1}(c_{n}\in s)( a_{u}(s)-a_{u}(\text{Occ}_{c_{n}}(s))) \tag{6}\] where \(Z\) is a normalizing factor, and \(a_{u}\) indicates the mean activation of unit \(u\), \(\text{Occ}_{c_{n}}(s)\) is a sentence \(s\) where candidate concept \(c_{n}\) is occluded, and \(\mathbb{1}(c_{n}\in s)\) is an indicator of whether \(c_{n}\) is included in the sentence \(s\). In short, the DoA measures how much a candidate concept contributes to the activation of the unit's top \(K\) sentences. If a candidate concept \(c_{n}\) appears in the topsentences \(\mathcal{S}\) and greatly reduces the activation of unit \(u\), then \(\mathsf{DoA}_{u,c_{n}}\) gets large, implying that the \(c_{n}\) is strongly aligned to unit \(u\).

Since each unit works as a concept detector whose activation value increases with the length of the input sentence (Section 4.2), it is essential to normalize the length of input for fair comparison of DoA values between the concepts that have different lengths one another. Without the length-normalization (_i.e_. each input sentence consists of just one instance of the candidate concept), the DoA metric has a bias to prefer lengthy concepts (_e.g_. phrases) because they typically have more signals that affect the unit activation than short candidate concepts (_e.g_. single words). In this work, we trained a ByteNet for the translation tasks and a VDCNN for the classification tasks, both to analyze properties of representations for language.

In our experiments, we analyze representations of each convolutional block layer.

Although \(M\) is used as a threshold to set how many concepts per unit are considered, different \(M\) values have little influence on quantitative results such as selectivity in Section 4.2. Figure 8 shows the mean and variance of selectivity values with different \(M=[1,3,5,10]\), where there is little variants in the overall trend; the sensitivity of the _replicate_ set is the highest, and that of _one instance_ is runner-up, and that of _random_ is the lowest. Whereas some units are sensitive to specific natural language concepts as shown in Section 4.3, other unites are not sensitive to any concepts at all.

We first define the unit interpretability for unit \(u\) as follows: \[\mathsf{Interpretability}(u)=\begin{cases}1&\text{if max}_{s\in\mathcal{S}}\{a _{u}(s)\}<\text{max}_{i=1,\dots,N}\{r_{i}\}\\ 0&\text{otherwise}\end{cases} \tag{7}\] where \(\mathcal{S}\) is the set of training sentences, \(a_{u}(s)\) is the activation value of unit \(u\), and \(r_{i}\) is the activation value of the sentence that is made up of replicating concept \(c_{i}\). We define unit \(u\) as interpretable when its \(\mathsf{Interpretability}(u)\) equals to 1, and otherwise as non-interpretable. The intuition is that if a replicated sentence that is composed of only one concept has a less activation value than the top-activated sentences, the unit is not sensitive to the concept compared to a sequence of different words.

For example, in unit 001 in the left of Figure 10, we discover that sentence structure involves many _commas_ in top activated sentences.

We introduce some units whose concepts have the shared meaning in Section 4.3. We here refer _concept cluster_ to the concepts that are aligned to the same unit and have similar semantics or grammatical roles. We analyze how clusters are formed in the units and how they vary with the target task and layer depth. Each central heat map represents the number of times each concept pair is aligned to the same unit. Since the concepts in the x, y-axes are ordered by the clustering result, if the diagonal blocks (concept clusters) emerge more strongly, the concepts in the same unit are more likely to have the similar meanings. In Figure 11, the units learned in the classification tasks tend to have stronger concept clusters than those learned in the translation tasks. Particularly, the concept clusters are highly evident in the units learned in DBpedia and AG News dataset. Our intuition is that units might have more benefits to solve the task by clustering similar concepts in the classification than the translation.

We analyze how concept clusters change by layer in each task.

We project each concept to the vector space using the three pretrained embeddings: (1) Glove (Pennington et al., 2014), (2) ConceptNet (Speer et al., 2017), (3) fastText.

Consistent with our intuition, in all datasets, both the document frequency and the delta of expected loss are closely related to the number of units per concept. It concludes that the representations are learned for identifying not only the frequent concepts in the training set and but also the important concepts for solving the target task. In section 4.4, we visualized how concepts are distributed across layers, where the model is trained on AG News dataset and English-to-German Europarl dataset.

In the classification tasks, we expect to find more concepts that are directly related to predicting the output label, as opposed to the translation tasks where the representations may have to include information on most of the words for an accurate translation. While our goal is not to relate each concept to one of the labels, we find several concepts that are more predictive to a particular label than others. Consistent with section 4.4, there are data-specific and task-specific concepts aligned in each layer; _i.e._ {_worst, 2 stars, awful_} at Yelp Review, {_film, ship, school_} at DBpedia, and some key words at translation datasets. Note that Yelp Review and DBpedia is a classification dataset, where the model is required to predict the polarity (_i.e._ +1 or -1) or ontology (_i.e._ Company, Educational Institution, Artist, Athlete, Officeholder, Mean of Transportation, Building, Natural Place, Village, Animal, Plant, Album, Film, Written Work) for given sentence in supervised setting.

Input: Figure 10: Examples of non-interpretable units, their concepts and top 5 activated sentences. Units are from representations learned on english-to-french translation dataset. See section D for details.

Input: Figure 10: Examples of non-interpretable units, their concepts and top 5 activated sentences. Units are from representations learned on english-to-french translation dataset. See section D for details.

Input: Figure 1: We discover the most activated sentences and aligned concepts to the units in hidden representations of deep convolutional networks. Aligned concepts appear frequently in most activated sentences, implying that those units respond selectively to specific natural language concepts.

Input: Figure 14: The Pearson correlations between the number of units per concept and (i) document frequency (top row), (ii) delta of expected loss (bottom row). They are measured at the final layer representation.

Input: Figure 15: 30 concepts selected by the number of aligned units in four encoding layers in VDCNN learned on Yelp Review dataset and DBpedia ontology dataset, and ByteNet learned on the English-to-German, English-to-French, and English-to-Czech parallel corpus. [#] symbol denotes morpheme concept.

Input: Figure 12: Averaged pairwise distances of concepts in each layer per task. Top and bottom row show the concept distances of translation models and classification models, respectively. For projecting concepts into the embedding space, we use three pretrained embedding: Glove, ConceptNet, FastText. See Appendix E.2 for more details.

Input: Figure 13: The number of unique concepts in each layer. It increases with the layer depth, which implies that the units in a deeper layer represent more diverse concepts.

Input: Figure 16: Aligned concepts per each task and their number of occurrences over multiple layers. See Appendix H for more details.

Input: Figure 11: Concept clusters of the last layer representations learned in each task. The more distinct the diagonal blocks are, the stronger the tendency that concepts aligned to the same unit share similar meanings or semantics. See Appendix E for details.

Input: Figure 8: Mean and variance of selectivity values for different M = [1, 3, 5, 10], where M is the number of selected concepts per unit. In all settings, the selectivity of replicate ones is the highest, that of one instance ones is runner-up, and that of random is the lowest near 0.

Input: Figure 7: Correlations between the number of units per concept and (a) document frequency and (b) delta of expected loss. Pearson correlation coefficients are measured at the final layer of the AG News sentence classification task. Some concepts aligned to many units are annotated. Results on other tasks are available in Appendix F.

Input: Figure 11: Concept clusters of the last layer representations learned in each task. The more distinct the diagonal blocks are, the stronger the tendency that concepts aligned to the same unit share similar meanings or semantics. See Appendix E for details.

Input: Figure 4: 30 concepts selected by the number of aligned units in three encoding layers of ByteNet learned on the Europarl translation dataset (left column) and VDCNN learned on AG-News (right column). [#] symbol denotes morpheme concepts. See section 4.4 for details.

Input: Figure 3: Examples of top activated sentences and aligned concepts to some units in several encoding layers of ByteNet and VDCNN. For each unit, concepts and their presence in top K sentences are shown in the same color. [#] symbol denotes morpheme concepts. See section 4.3 for details.

Input: Figure 14: The Pearson correlations between the number of units per concept and (i) document frequency (top row), (ii) delta of expected loss (bottom row). They are measured at the final layer representation.

Input: Figure 15: 30 concepts selected by the number of aligned units in four encoding layers in VDCNN learned on Yelp Review dataset and DBpedia ontology dataset, and ByteNet learned on the English-to-German, English-to-French, and English-to-Czech parallel corpus. [#] symbol denotes morpheme concept.

Input: Figure 12: Averaged pairwise distances of concepts in each layer per task. Top and bottom row show the concept distances of translation models and classification models, respectively. For projecting concepts into the embedding space, we use three pretrained embedding: Glove, ConceptNet, FastText. See Appendix E.2 for more details.

Input: Figure 13: The number of unique concepts in each layer. It increases with the layer depth, which implies that the units in a deeper layer represent more diverse concepts.

Input: Figure 16: Aligned concepts per each task and their number of occurrences over multiple layers. See Appendix H for more details.

Input: Figure 2: Mean and variance of selectivity values over all units in the learned representation for each dataset. Sentences including the concepts that our alignment method discovers always activate units significantly more than random sentences. See section 4.2 for details.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, I'll assess the claim "What natural language concepts are captured by each unit in the representations learned from NLP tasks?"

The evidence strongly supports this claim through rigorous methodology and comprehensive analysis. The researchers:
- Developed methods to identify and validate concepts captured by individual units
- Found units that selectively respond to specific linguistic patterns (morphemes, words, phrases)
- Demonstrated units capturing both syntactic and semantic concepts
- Validated findings across multiple tasks and architectures
- Provided quantitative metrics and visualizations
- Acknowledged limitations and areas needing further research

The claim is well-stated as it accurately reflects what the study investigated and found, without overgenerating beyond the evidence. The findings are thoroughly supported by detailed methodology and results.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_What natural language concepts are captured by each unit in the representations learned from NLP tasks?

The review comment to be evaluated is:

The paper is well written and structured, presenting the problem clearly and accurately. It contains considerable relevant references and enough background knowledge. It nicely motivates the proposed approach, locates the contributions in the state-of-the-art and reviews related work. It is also very honest in terms of how it differs on the technical level from existing approaches.
The paper presents interesting and novel findings to further state-of-the-art’s understanding on how language concepts are represented in the intermediate layers of deep convolutional neural networks, showing that channels in convolutional representations are selectively sensitive to specific natural language concepts. It also nicely discusses how concepts granularity evolves with layers’ deepness in the case of natural language tasks.
What I am missing, however, is an empirical study of concepts coverage over multiple layers, studying the multiple occurrences of single concepts at different layers, and a deeper dive on the rather noisy elements of natural language and the layers’ activation dynamics towards such elements.
Overall, however, the ideas presented in the paper are interesting and original, and the experimental section is convincing. My recommendation is to accept this submission.

The evidence to be evaluated is:Recent works on interpreting hidden representations at unit-level were mostly motivated by their counterparts in computer vision.

In the NLP domain, studies including Karpathy et al. (2015); Tang et al. (2017); Qian et al. (2016); Shi et al. (2016) analyzed the internal mechanisms of deep models used for NLP and found intriguing properties that appear in units of hidden representations.

Compared with these previous studies, we focus on discovering a much wider variety of _natural language concepts_, including any morphemes, words, and phrases all found in the training data. To the best our knowledge, this is the first attempt to discover concepts among all that exist in the form of natural language from the training corpus. By extending the scope of detected concepts to meaningful building blocks of natural language, we provide insights into how various linguistic features are encoded by the hidden units of deep representations. Most previous work that analyzes the learned representation of NLP tasks focused on constructing downstream tasks that predict concepts of interest. A common approach is to measure the performance of a classification model that predicts the concept of interest to see whether those concepts are encoded in representation of a input sentence. For example, Conneau et al. (2018); Adi et al. (2017); Zhu et al. (2018) proposed several probing tasks to test whether the (non-)linear regression model can predict well the syntactic or semantic information from the representation learned on translation tasks or the skip-thought or word embedding vectors. Shi et al. (2016); Belinkov et al. (2017) constructed regression tasks that predict labels such as voice, tense, part-of-speech tag, and morpheme from the encoder representation of the learned model in translation task.

(1) By identifying the role of the individual units, rather than analyzing the representation as a whole, we provide more fine-grained understanding of how the representations encode informative factors in training data. (2) Rather than limiting the linguistic features within the representation to be discovered, we focus on covering concepts of fundamental building blocks of natural language (morphemes, words, and phrases) present in the training data, providing more flexible interpretation results without relying on a predefined set of concepts.

We first train a CNN model for each natural language task (_e.g._ translation and classification) and retrieve training sentences that highly activate specific units. Interestingly, we discover morphemes, words and phrases that appear dominantly within these retrieved sentences, implying that those concepts have a significant impact on the activation value of the unit. Then, we find a set of concepts which attribute a lot to the unit activation by measuring activation value of each replicated candidate concept, and align them to unit.

Interestingly, some natural language patterns such as morphemes, words, phrases frequently appear in the retrieved sentences (see Figure 1), implying that those concepts might have a large attribution to the activation value of that unit.

For constructing candidate concepts, we parse each of top \(K\) sentences with a constituency parser (Kitaev and Klein, 2018). Within the constituency-based parse tree, we define candidate concepts as all terminal and non-terminal nodes (_e.g._ from sentence _John hit the balls_, we obtain candidate concepts as {_John, hit, the balls, hit the balls, John hit the balls_}). We also break each word into morphemes using a morphological analysis tool (Virpioja et al., 2013) and add them to candidate concepts (_e.g._ from word _balls_, we obtain morphemes {_ball, s_}). We repeat this process for every top \(K\) sentence and build a set of candidate concepts for unit \(u\), which is denoted as \(\mathcal{C}_{u}=\{c_{1},...,c_{N}\}\), where \(N\) is the number of candidate concepts of the unit.

Finally, the degree of alignment (DoA) between a candidate concept \(c_{n}\) and a unit \(u\) is defined as follows: \[\mathsf{DoA}_{u,c_{n}}=a_{u}(r_{n}) \tag{1}\] In short, the \(\mathsf{DoA}\)1 measures the extent to unit \(u\)'s activation is sensitive to the presence of candidate concept \(c_{n}\). If a candidate concept \(c_{n}\) appears in the top \(K\) sentences and unit's activation value \(a_{u}\) is responsive to \(c_{n}\) a lot, then \(\mathsf{DoA}_{u,c_{n}}\) gets large, suggesting that candidate concept \(c_{n}\) is strongly aligned to unit \(u\).

Finally, for each unit \(u\), we define a set of its aligned concepts \(\mathcal{C}_{u}^{*}=\{c_{1}^{*},...,c_{M}^{*}\}\) as \(M\) candidate concepts with the largest DoA values in \(C_{u}\). Depending on how we set \(M\), we can detect different numbers of concepts per unit.

We then focus on the representations in each encoder layer of ByteNet and convolutional layer of VDCNN, because as Mou et al. (2016) pointed out, the representation of the decoder (the output layer in the case of classification) is specialized for predicting the output of the target task rather than for learning the semantics of the input text. To quantitatively evaluate how well our approach aligns concepts, we measure how selectively each unit responds to the aligned concept. Motivated by Morcos et al. (2018), we define the **concept selectivity** of a unit \(u\), to a set of concepts \(\mathcal{C}_{u}^{*}\) that our alignment method detects, as follows: \[\mathsf{Sel}_{u}=\frac{\mu_{+}-\mu_{-}}{\max_{s\in\mathcal{S}}a_{u}(s)-\min_{s \in\mathcal{S}}a_{u}(s)} \tag{2}\] where \(\mathcal{S}\) denotes all sentences in training set, and \(\mu_{+}=\frac{1}{|\mathcal{S}_{+}|}\sum_{s\in\mathcal{S}_{+}}a_{u}(s)\) is the average value of unit activation when forwarding a set of sentences \(\mathcal{S}_{+}\), which is defined as one of the following: * _replicate_: \(\mathcal{S}_{+}\) contains the sentences created by replicating each concept in \(\mathcal{C}_{u}^{*}\). As before, the sentence length is set as the average length of all training sentences for fair comparison.

More interestingly, we discover that many units could capture specific meanings or syntactic roles beyond superficial, low-level patterns. For example, unit 690 of the 14th layer in ByteNet captures (_what, who, where_) concepts, all of which play a similar grammatical role. On the other hand, unit 224 of the 14th layer in ByteNet and unit 53 of the 0th layer in VDCNN each captures semantically similar concepts, with the ByteNet unit detecting the meaning of certainty in knowledge (_sure, know, aware_) and the VDCNN unit detecting years (_1999, 1969, 1992_). This suggests that, although we train character-level CNNs with feeding sentences as the form of discrete symbols (_i.e._ character indices), individual units could capture natural language concepts sharing a similar semantic or grammatical role.

We note that there are units that detect concepts more abstract than just morphemes, words, or phrases, and for these units, our method tends to align relevant lower-level concepts. For example, in unit 244 of the 3rd layer in VDCNN, while each aligned concept emerges only once in the top \(K\) sentences, all top \(K\) sentences have similar _nucances_ like positive sentiments. In this case, our method does capture relevant phrase-level concepts (e.g., _very disappointing, absolute worst place_), indicating that the higher-level _nucance_ (e.g., negativity) is indirectly captured.

* Several units in the CNN learned on NLP tasks respond selectively to specific natural language concepts, rather than getting activated in an uninterpretable way. This means that these units can serve as detectors for specific natural language concepts.

* There are units capturing syntactically or semantically related concepts, suggesting that they model the _meaning or grammatical role shared between those concepts_, as opposed to superficially modeling each natural language _symbol_.

Overall, we find that data and task-specific concepts are likely to be aligned to many units. In AG News, since the task is to classify given sentences into following categories; _World_, _Sports_, _Business_ and _Science/Tech_, concepts related to these topics commonly emerge. Similarly, we can see that units learned for Europarl dataset focus to encode some key words (_e.g. vote_, _propose_, _environment_) in the training corpus.

To confirm that it also holds for representations learned in NLP tasks, we divide granularity of natural language concepts to the morpheme, word and \(N\)-gram phrase (\(N=2,3,4,5\)), and observe the number of units that they are aligned in different layers.

In deeper models, this may suggest that the middle layer's representation may be already informative enough to encode the input text, and our result may partly coincide with that of Mou et al. (2016), which shows that representation of intermediate layers is more transferable than that of deeper layers in language tasks, unlike in computer vision where deeper layers are usually more useful and discriminative.

We observe that the concepts do not appear uniformly; some concepts are aligned to many units, while others are aligned to few or even no units.

First, the concepts with a higher frequency in training data may be aligned to more units. Figure 7-(a) shows the correlation between the frequency of each concept in the training corpus and the number of units where each concept is aligned in the last layer of the topic classification model learned on AG News dataset. Second, the concepts that have more influence on the objective function (expected loss) may be aligned to more units.

Another direction is to quantify the properties of individual units in other models widely used in NLP tasks. In particular, combining our definition of concepts with the attention mechanism (_e.g._ Bahdanau et al. (2015)) could be a promising direction, because it can reveal how the representations are attended by the model to capture concepts, helping us better understand the decision-making process of popular deep models.

In section 3.2, we define Degree of Alignment (DoA) between concept \(c_{n}\) and unit \(u\) as activation value of unit \(u\) for replication of \(c_{n}\).

We then define DoA between candidate concept \(c_{n}\) and unit \(u\) by using PMI as follow: \[\text{DoA}_{u,c_{n}}=\text{pmi}(u,c_{n})=\log\frac{p(u,c_{n})}{p(u)p(c_{n})}, \text{where} \tag{5a}\] \[p(u)=\frac{\#[s\in\mathcal{S},s\in\text{topK}(u)]}{\#[s\in \mathcal{S}]},\] (5b) \[p(c_{n})=\frac{\#[s\in\mathcal{S},c_{n}\in s]}{\#[s\in \mathcal{S}]},\] (5c) \[p(u,c_{n})=\frac{\#[s\in\text{topK}(u),c_{n}\in s]}{\#[s\in \text{topK}(u)]} \tag{5d}\]

We tested concept alignments with the following concept occlusion method. For each of the top \(K\) sentences, we replace a \(c_{n}\) by dummy character tokens which have no meaning, forward it to the model, and measure the reduction of the unit activation value. We repeat this for every candidate concept in the sentences - as a result, we can identify which candidate concept greatly reduce unit activation values. We thus define concepts aligned to each unit as the candidate concept that consistently lower the unit activation across the top \(K\) sentences. More formally, for each unit \(u\), let \(\mathcal{S}=\{s_{1},...,s_{K}\}\) be top \(K\) activated sentences. Since we occlude each candidate concept in sentences, we define the set of candidate concept \(\mathcal{C}=\{c_{1},...,c_{N}\}\), obtained from parsing each sentence in \(\mathcal{S}\). We define the **degree of alignment (DoA)** between a concept \(c\in\mathcal{C}\) and a unit \(u\) as: \[\text{DoA}_{u,c_{n}}=\frac{1}{Z}\sum_{s\in\mathcal{S}}\mathbb{1}(c_{n}\in s)( a_{u}(s)-a_{u}(\text{Occ}_{c_{n}}(s))) \tag{6}\] where \(Z\) is a normalizing factor, and \(a_{u}\) indicates the mean activation of unit \(u\), \(\text{Occ}_{c_{n}}(s)\) is a sentence \(s\) where candidate concept \(c_{n}\) is occluded, and \(\mathbb{1}(c_{n}\in s)\) is an indicator of whether \(c_{n}\) is included in the sentence \(s\). In short, the DoA measures how much a candidate concept contributes to the activation of the unit's top \(K\) sentences. If a candidate concept \(c_{n}\) appears in the topsentences \(\mathcal{S}\) and greatly reduces the activation of unit \(u\), then \(\mathsf{DoA}_{u,c_{n}}\) gets large, implying that the \(c_{n}\) is strongly aligned to unit \(u\).

Since each unit works as a concept detector whose activation value increases with the length of the input sentence (Section 4.2), it is essential to normalize the length of input for fair comparison of DoA values between the concepts that have different lengths one another. Without the length-normalization (_i.e_. each input sentence consists of just one instance of the candidate concept), the DoA metric has a bias to prefer lengthy concepts (_e.g_. phrases) because they typically have more signals that affect the unit activation than short candidate concepts (_e.g_. single words). In this work, we trained a ByteNet for the translation tasks and a VDCNN for the classification tasks, both to analyze properties of representations for language.

In our experiments, we analyze representations of each convolutional block layer.

Although \(M\) is used as a threshold to set how many concepts per unit are considered, different \(M\) values have little influence on quantitative results such as selectivity in Section 4.2. Figure 8 shows the mean and variance of selectivity values with different \(M=[1,3,5,10]\), where there is little variants in the overall trend; the sensitivity of the _replicate_ set is the highest, and that of _one instance_ is runner-up, and that of _random_ is the lowest. Whereas some units are sensitive to specific natural language concepts as shown in Section 4.3, other unites are not sensitive to any concepts at all.

We first define the unit interpretability for unit \(u\) as follows: \[\mathsf{Interpretability}(u)=\begin{cases}1&\text{if max}_{s\in\mathcal{S}}\{a _{u}(s)\}<\text{max}_{i=1,\dots,N}\{r_{i}\}\\ 0&\text{otherwise}\end{cases} \tag{7}\] where \(\mathcal{S}\) is the set of training sentences, \(a_{u}(s)\) is the activation value of unit \(u\), and \(r_{i}\) is the activation value of the sentence that is made up of replicating concept \(c_{i}\). We define unit \(u\) as interpretable when its \(\mathsf{Interpretability}(u)\) equals to 1, and otherwise as non-interpretable. The intuition is that if a replicated sentence that is composed of only one concept has a less activation value than the top-activated sentences, the unit is not sensitive to the concept compared to a sequence of different words.

For example, in unit 001 in the left of Figure 10, we discover that sentence structure involves many _commas_ in top activated sentences.

We introduce some units whose concepts have the shared meaning in Section 4.3. We here refer _concept cluster_ to the concepts that are aligned to the same unit and have similar semantics or grammatical roles. We analyze how clusters are formed in the units and how they vary with the target task and layer depth. Each central heat map represents the number of times each concept pair is aligned to the same unit. Since the concepts in the x, y-axes are ordered by the clustering result, if the diagonal blocks (concept clusters) emerge more strongly, the concepts in the same unit are more likely to have the similar meanings. In Figure 11, the units learned in the classification tasks tend to have stronger concept clusters than those learned in the translation tasks. Particularly, the concept clusters are highly evident in the units learned in DBpedia and AG News dataset. Our intuition is that units might have more benefits to solve the task by clustering similar concepts in the classification than the translation.

We analyze how concept clusters change by layer in each task.

We project each concept to the vector space using the three pretrained embeddings: (1) Glove (Pennington et al., 2014), (2) ConceptNet (Speer et al., 2017), (3) fastText.

Consistent with our intuition, in all datasets, both the document frequency and the delta of expected loss are closely related to the number of units per concept. It concludes that the representations are learned for identifying not only the frequent concepts in the training set and but also the important concepts for solving the target task. In section 4.4, we visualized how concepts are distributed across layers, where the model is trained on AG News dataset and English-to-German Europarl dataset.

In the classification tasks, we expect to find more concepts that are directly related to predicting the output label, as opposed to the translation tasks where the representations may have to include information on most of the words for an accurate translation. While our goal is not to relate each concept to one of the labels, we find several concepts that are more predictive to a particular label than others. Consistent with section 4.4, there are data-specific and task-specific concepts aligned in each layer; _i.e._ {_worst, 2 stars, awful_} at Yelp Review, {_film, ship, school_} at DBpedia, and some key words at translation datasets. Note that Yelp Review and DBpedia is a classification dataset, where the model is required to predict the polarity (_i.e._ +1 or -1) or ontology (_i.e._ Company, Educational Institution, Artist, Athlete, Officeholder, Mean of Transportation, Building, Natural Place, Village, Animal, Plant, Album, Film, Written Work) for given sentence in supervised setting.

Input: Figure 10: Examples of non-interpretable units, their concepts and top 5 activated sentences. Units are from representations learned on english-to-french translation dataset. See section D for details.

Input: Figure 10: Examples of non-interpretable units, their concepts and top 5 activated sentences. Units are from representations learned on english-to-french translation dataset. See section D for details.

Input: Figure 1: We discover the most activated sentences and aligned concepts to the units in hidden representations of deep convolutional networks. Aligned concepts appear frequently in most activated sentences, implying that those units respond selectively to specific natural language concepts.

Input: Figure 14: The Pearson correlations between the number of units per concept and (i) document frequency (top row), (ii) delta of expected loss (bottom row). They are measured at the final layer representation.

Input: Figure 15: 30 concepts selected by the number of aligned units in four encoding layers in VDCNN learned on Yelp Review dataset and DBpedia ontology dataset, and ByteNet learned on the English-to-German, English-to-French, and English-to-Czech parallel corpus. [#] symbol denotes morpheme concept.

Input: Figure 12: Averaged pairwise distances of concepts in each layer per task. Top and bottom row show the concept distances of translation models and classification models, respectively. For projecting concepts into the embedding space, we use three pretrained embedding: Glove, ConceptNet, FastText. See Appendix E.2 for more details.

Input: Figure 13: The number of unique concepts in each layer. It increases with the layer depth, which implies that the units in a deeper layer represent more diverse concepts.

Input: Figure 16: Aligned concepts per each task and their number of occurrences over multiple layers. See Appendix H for more details.

Input: Figure 11: Concept clusters of the last layer representations learned in each task. The more distinct the diagonal blocks are, the stronger the tendency that concepts aligned to the same unit share similar meanings or semantics. See Appendix E for details.

Input: Figure 8: Mean and variance of selectivity values for different M = [1, 3, 5, 10], where M is the number of selected concepts per unit. In all settings, the selectivity of replicate ones is the highest, that of one instance ones is runner-up, and that of random is the lowest near 0.

Input: Figure 7: Correlations between the number of units per concept and (a) document frequency and (b) delta of expected loss. Pearson correlation coefficients are measured at the final layer of the AG News sentence classification task. Some concepts aligned to many units are annotated. Results on other tasks are available in Appendix F.

Input: Figure 11: Concept clusters of the last layer representations learned in each task. The more distinct the diagonal blocks are, the stronger the tendency that concepts aligned to the same unit share similar meanings or semantics. See Appendix E for details.

Input: Figure 4: 30 concepts selected by the number of aligned units in three encoding layers of ByteNet learned on the Europarl translation dataset (left column) and VDCNN learned on AG-News (right column). [#] symbol denotes morpheme concepts. See section 4.4 for details.

Input: Figure 3: Examples of top activated sentences and aligned concepts to some units in several encoding layers of ByteNet and VDCNN. For each unit, concepts and their presence in top K sentences are shown in the same color. [#] symbol denotes morpheme concepts. See section 4.3 for details.

Input: Figure 14: The Pearson correlations between the number of units per concept and (i) document frequency (top row), (ii) delta of expected loss (bottom row). They are measured at the final layer representation.

Input: Figure 15: 30 concepts selected by the number of aligned units in four encoding layers in VDCNN learned on Yelp Review dataset and DBpedia ontology dataset, and ByteNet learned on the English-to-German, English-to-French, and English-to-Czech parallel corpus. [#] symbol denotes morpheme concept.

Input: Figure 12: Averaged pairwise distances of concepts in each layer per task. Top and bottom row show the concept distances of translation models and classification models, respectively. For projecting concepts into the embedding space, we use three pretrained embedding: Glove, ConceptNet, FastText. See Appendix E.2 for more details.

Input: Figure 13: The number of unique concepts in each layer. It increases with the layer depth, which implies that the units in a deeper layer represent more diverse concepts.

Input: Figure 16: Aligned concepts per each task and their number of occurrences over multiple layers. See Appendix H for more details.

Input: Figure 2: Mean and variance of selectivity values over all units in the learned representation for each dataset. Sentences including the concepts that our alignment method discovers always activate units significantly more than random sentences. See section 4.2 for details.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, this claim about natural language concepts captured by units in NLP representations is well-stated and supported by thorough empirical analysis. The research demonstrates that individual units can selectively detect specific language concepts (morphemes, words, phrases) and even capture semantic/syntactic relationships. The evidence includes detailed methodological explanations, quantitative metrics (selectivity, DoA), visualization of concept clusters, and multiple validation approaches. The review comment confirms the paper's strong empirical foundation while noting some limitations around concept coverage analysis. The claim appropriately reflects the scope and certainty of the findings without overreaching.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_What natural language concepts are captured by each unit in the representations learned from NLP tasks?

The review comment to be evaluated is:

========== Edit following authors' response  ==========

Thank you for your detailed response and updated version. I think the new revision is significantly improved, mainly in more quantitative analyses and details in several places. I have updated my evaluation accordingly.

See a few more points below.

1. Thank you for clarifying your definition of concepts. I still think that the word "concept" has a strong semantic connotation, while the linguistic elements your analyses capture may do other things. The results in appendix E do show that some semantic clusters arise. It's especially interesting to see the blocks in some of the heat maps, where similar "concepts" are clustered together (like the sports terms in AG); consider commenting on this.

2. The new quantitative analyses are helpful. One other suggestion that I mentioned before is to connect detected concepts to external resources like WordNet or ConceptNet. That would help show that "concepts" are indeed semantic objects.

3. The motivation for replicating as normalizing for length does make sense, although the input would still be unnatural. The comparison to "one instance" is helpful, but it's interesting that the differences between it and replication in figure 2 are not large. It would be good to show results that substantiate your assumption that without replication there will be a bias towards lengthy concepts. Does "one instance" detect more lengthy concepts than replication?

4. The results on frequency and loss difference in 4.5 are very interesting. There is another angle to consider frequency: words that appear frequently often carry less semantic content (e.g. function words), so one might conjecture that they would require less units. It may be interesting to look at which concepts are detected at each frequency bin.

5. Minor points: section 2.2 still mentions "regression" where it should be "classification".

6. A few remaining grammar issues:
- "one concept has a less activation value.." - rephrase
- end of section 3.3: "this experiments" -> "these experiments"

========== Original review follows ==========

Summary:
=======
This paper analyzes individual units in CNN models for text classification and translation tasks. It defines a measure of sensitivity for a unit and evaluates how sensitive each unit is to "concepts" in the input text, where concepts are morphemes, words, and phrases. The analysis shows that some units seem to learn semantic concepts, while others capture linguistic elements that are frequent or relevant for the end task. Layer-wise results show some correspondence between layer depth and linguistic element size.

The paper studies an important question that is relatively under-studies in NLP compared to the computer vision community. The motivation for the work is quite convincing.
I found some of the results and analysis interesting, but overall felt that the work can be made much stronger by more quantitative evaluations. I am also worried that the notion of "concept" is misleading here. See below for this and other comments. I am willing to reconsider my evaluation pending response to the below issues.

Main comments:
=============
1. Concepts:
- morphemes, words, and phrases - are these "concepts"? They are indeed "fundamental building blocks of natural language" (2.2), but "concepts" has a more semantic connotation that I'm not sure these units target at.
- Some of the results do suggest that units learn concepts, as the analysis in 4.2 shows a "unit detecting the meaning of certainty in knowledge" and later units that have similar sentiments. It would be informative to quantify this in some way, for example by matching detected concepts to WordNet synsets, sentiment lexicons, etc., or else tagging and classifying them with various NLP tools. This could also reveal if units learn more syntactic or semantic concepts, and so on.
2. Generally, many of the analyses in the paper are qualitative and on a small scale. The results will be more convincing with more automatic aggregate measures.
3. The structure of the paper is confusing. Section3 starts with the approach but then mentions datasets and tasks (3.1). Section 4 is titled experiments, but section 4.1 starts with defining the concept selectivity. I would suggest reorganizing sections 3 and 4, such that section 3 describes all the methods and metrics, while dataset-specific parts are moved to section 4.
4. section 3.2 should provide more details on the sentence representation and how its obtained in the CNN models. A mathematical derivation and/or figure could be helpful. It is also not clear to me what's the motivation for mean-pooling over the l entries of the vector.
5. section 3.3: the use of replicated text for "concept alignment" is puzzling. This is not a natural input to the model, and I think more justification and motivation åre needed for this issue, as well as perhaps comparison with other approaches.
6. I found section 4.4 very interesting. It shows some intuitive results of larger linguistic elements learned at higher layers, but then some results that do not show such a trend. Then, hypothesizing that the middle layers are sufficient AND validating the hypothesis by retraining the model is excellent. It's a very nice demonstration that the analysis can lead to model improvements.
7. Figure 2 seems to be almost caused by construction of the different options for S_+. Is it surprising that the replicate set has the highest sensitivity? Is there a better control setup than comparing with a random set?
8. One concern that I have is the effect of confounding factors like frequency on the results. The papers occasionally attributes importance to concepts (e.g. in 4.2), but I wonder if instead we may be seeing more frequent words. Controlling for the effect of frequency would be useful.

Minor comments:
==============
- Section 2.2, first paragraph: regression should be changed to classification
- The related work is generally relevant, although one could mention a few other papers that analyzed individual neurons in NLP tasks [1, 2]
- section 4.1: the random set may perhaps be denoted by something more neutral, not S_+ as the replicate and inclusion sets.
- section 4.3, last paragraph: listing examples showing that units in Europarl focus on key words would be good.
- Figure 5, y axis label: should this be number of units instead of concepts?
- Appendix A has several interesting points but there is no reference to them from the main paper.

Writing, grammar, etc.:
======================
- Introduction: among them - who is them?
- 2.1: motivated from -> motivated by; In computer vision community -> In the computer vision community
- 2.1: quantifying characteristics of representations in layer-wise -> rephrase
- 3.2: dimension of sentence -> dimension of the/a sentence
- 4.1: to which -> remove "which"
- 4.2: in the several encoding layer -> in several encoding layers
- 4.3: aliged -> aligned
- Capitalize titles in references
- A.2: with following -> with the following; how much candidate -> how much a candidate; consider following -> consider the following
- A.3: induces similar bias -> induces a bias; such phrase -> such a phrase; on very -> on a very
- C: where model -> where the model; In consistent -> Consistent; where model -> where the model

References
==========
[1] Qian et al., Analyzing linguistic knowledge in sequential model of sentence
[2] Shi et al., Why Neural Translations are the Right Length

The evidence to be evaluated is:Recent works on interpreting hidden representations at unit-level were mostly motivated by their counterparts in computer vision.

In the NLP domain, studies including Karpathy et al. (2015); Tang et al. (2017); Qian et al. (2016); Shi et al. (2016) analyzed the internal mechanisms of deep models used for NLP and found intriguing properties that appear in units of hidden representations.

Compared with these previous studies, we focus on discovering a much wider variety of _natural language concepts_, including any morphemes, words, and phrases all found in the training data. To the best our knowledge, this is the first attempt to discover concepts among all that exist in the form of natural language from the training corpus. By extending the scope of detected concepts to meaningful building blocks of natural language, we provide insights into how various linguistic features are encoded by the hidden units of deep representations. Most previous work that analyzes the learned representation of NLP tasks focused on constructing downstream tasks that predict concepts of interest. A common approach is to measure the performance of a classification model that predicts the concept of interest to see whether those concepts are encoded in representation of a input sentence. For example, Conneau et al. (2018); Adi et al. (2017); Zhu et al. (2018) proposed several probing tasks to test whether the (non-)linear regression model can predict well the syntactic or semantic information from the representation learned on translation tasks or the skip-thought or word embedding vectors. Shi et al. (2016); Belinkov et al. (2017) constructed regression tasks that predict labels such as voice, tense, part-of-speech tag, and morpheme from the encoder representation of the learned model in translation task.

(1) By identifying the role of the individual units, rather than analyzing the representation as a whole, we provide more fine-grained understanding of how the representations encode informative factors in training data. (2) Rather than limiting the linguistic features within the representation to be discovered, we focus on covering concepts of fundamental building blocks of natural language (morphemes, words, and phrases) present in the training data, providing more flexible interpretation results without relying on a predefined set of concepts.

We first train a CNN model for each natural language task (_e.g._ translation and classification) and retrieve training sentences that highly activate specific units. Interestingly, we discover morphemes, words and phrases that appear dominantly within these retrieved sentences, implying that those concepts have a significant impact on the activation value of the unit. Then, we find a set of concepts which attribute a lot to the unit activation by measuring activation value of each replicated candidate concept, and align them to unit.

Interestingly, some natural language patterns such as morphemes, words, phrases frequently appear in the retrieved sentences (see Figure 1), implying that those concepts might have a large attribution to the activation value of that unit.

For constructing candidate concepts, we parse each of top \(K\) sentences with a constituency parser (Kitaev and Klein, 2018). Within the constituency-based parse tree, we define candidate concepts as all terminal and non-terminal nodes (_e.g._ from sentence _John hit the balls_, we obtain candidate concepts as {_John, hit, the balls, hit the balls, John hit the balls_}). We also break each word into morphemes using a morphological analysis tool (Virpioja et al., 2013) and add them to candidate concepts (_e.g._ from word _balls_, we obtain morphemes {_ball, s_}). We repeat this process for every top \(K\) sentence and build a set of candidate concepts for unit \(u\), which is denoted as \(\mathcal{C}_{u}=\{c_{1},...,c_{N}\}\), where \(N\) is the number of candidate concepts of the unit.

Finally, the degree of alignment (DoA) between a candidate concept \(c_{n}\) and a unit \(u\) is defined as follows: \[\mathsf{DoA}_{u,c_{n}}=a_{u}(r_{n}) \tag{1}\] In short, the \(\mathsf{DoA}\)1 measures the extent to unit \(u\)'s activation is sensitive to the presence of candidate concept \(c_{n}\). If a candidate concept \(c_{n}\) appears in the top \(K\) sentences and unit's activation value \(a_{u}\) is responsive to \(c_{n}\) a lot, then \(\mathsf{DoA}_{u,c_{n}}\) gets large, suggesting that candidate concept \(c_{n}\) is strongly aligned to unit \(u\).

Finally, for each unit \(u\), we define a set of its aligned concepts \(\mathcal{C}_{u}^{*}=\{c_{1}^{*},...,c_{M}^{*}\}\) as \(M\) candidate concepts with the largest DoA values in \(C_{u}\). Depending on how we set \(M\), we can detect different numbers of concepts per unit.

We then focus on the representations in each encoder layer of ByteNet and convolutional layer of VDCNN, because as Mou et al. (2016) pointed out, the representation of the decoder (the output layer in the case of classification) is specialized for predicting the output of the target task rather than for learning the semantics of the input text. To quantitatively evaluate how well our approach aligns concepts, we measure how selectively each unit responds to the aligned concept. Motivated by Morcos et al. (2018), we define the **concept selectivity** of a unit \(u\), to a set of concepts \(\mathcal{C}_{u}^{*}\) that our alignment method detects, as follows: \[\mathsf{Sel}_{u}=\frac{\mu_{+}-\mu_{-}}{\max_{s\in\mathcal{S}}a_{u}(s)-\min_{s \in\mathcal{S}}a_{u}(s)} \tag{2}\] where \(\mathcal{S}\) denotes all sentences in training set, and \(\mu_{+}=\frac{1}{|\mathcal{S}_{+}|}\sum_{s\in\mathcal{S}_{+}}a_{u}(s)\) is the average value of unit activation when forwarding a set of sentences \(\mathcal{S}_{+}\), which is defined as one of the following: * _replicate_: \(\mathcal{S}_{+}\) contains the sentences created by replicating each concept in \(\mathcal{C}_{u}^{*}\). As before, the sentence length is set as the average length of all training sentences for fair comparison.

More interestingly, we discover that many units could capture specific meanings or syntactic roles beyond superficial, low-level patterns. For example, unit 690 of the 14th layer in ByteNet captures (_what, who, where_) concepts, all of which play a similar grammatical role. On the other hand, unit 224 of the 14th layer in ByteNet and unit 53 of the 0th layer in VDCNN each captures semantically similar concepts, with the ByteNet unit detecting the meaning of certainty in knowledge (_sure, know, aware_) and the VDCNN unit detecting years (_1999, 1969, 1992_). This suggests that, although we train character-level CNNs with feeding sentences as the form of discrete symbols (_i.e._ character indices), individual units could capture natural language concepts sharing a similar semantic or grammatical role.

We note that there are units that detect concepts more abstract than just morphemes, words, or phrases, and for these units, our method tends to align relevant lower-level concepts. For example, in unit 244 of the 3rd layer in VDCNN, while each aligned concept emerges only once in the top \(K\) sentences, all top \(K\) sentences have similar _nucances_ like positive sentiments. In this case, our method does capture relevant phrase-level concepts (e.g., _very disappointing, absolute worst place_), indicating that the higher-level _nucance_ (e.g., negativity) is indirectly captured.

* Several units in the CNN learned on NLP tasks respond selectively to specific natural language concepts, rather than getting activated in an uninterpretable way. This means that these units can serve as detectors for specific natural language concepts.

* There are units capturing syntactically or semantically related concepts, suggesting that they model the _meaning or grammatical role shared between those concepts_, as opposed to superficially modeling each natural language _symbol_.

Overall, we find that data and task-specific concepts are likely to be aligned to many units. In AG News, since the task is to classify given sentences into following categories; _World_, _Sports_, _Business_ and _Science/Tech_, concepts related to these topics commonly emerge. Similarly, we can see that units learned for Europarl dataset focus to encode some key words (_e.g. vote_, _propose_, _environment_) in the training corpus.

To confirm that it also holds for representations learned in NLP tasks, we divide granularity of natural language concepts to the morpheme, word and \(N\)-gram phrase (\(N=2,3,4,5\)), and observe the number of units that they are aligned in different layers.

In deeper models, this may suggest that the middle layer's representation may be already informative enough to encode the input text, and our result may partly coincide with that of Mou et al. (2016), which shows that representation of intermediate layers is more transferable than that of deeper layers in language tasks, unlike in computer vision where deeper layers are usually more useful and discriminative.

We observe that the concepts do not appear uniformly; some concepts are aligned to many units, while others are aligned to few or even no units.

First, the concepts with a higher frequency in training data may be aligned to more units. Figure 7-(a) shows the correlation between the frequency of each concept in the training corpus and the number of units where each concept is aligned in the last layer of the topic classification model learned on AG News dataset. Second, the concepts that have more influence on the objective function (expected loss) may be aligned to more units.

Another direction is to quantify the properties of individual units in other models widely used in NLP tasks. In particular, combining our definition of concepts with the attention mechanism (_e.g._ Bahdanau et al. (2015)) could be a promising direction, because it can reveal how the representations are attended by the model to capture concepts, helping us better understand the decision-making process of popular deep models.

In section 3.2, we define Degree of Alignment (DoA) between concept \(c_{n}\) and unit \(u\) as activation value of unit \(u\) for replication of \(c_{n}\).

We then define DoA between candidate concept \(c_{n}\) and unit \(u\) by using PMI as follow: \[\text{DoA}_{u,c_{n}}=\text{pmi}(u,c_{n})=\log\frac{p(u,c_{n})}{p(u)p(c_{n})}, \text{where} \tag{5a}\] \[p(u)=\frac{\#[s\in\mathcal{S},s\in\text{topK}(u)]}{\#[s\in \mathcal{S}]},\] (5b) \[p(c_{n})=\frac{\#[s\in\mathcal{S},c_{n}\in s]}{\#[s\in \mathcal{S}]},\] (5c) \[p(u,c_{n})=\frac{\#[s\in\text{topK}(u),c_{n}\in s]}{\#[s\in \text{topK}(u)]} \tag{5d}\]

We tested concept alignments with the following concept occlusion method. For each of the top \(K\) sentences, we replace a \(c_{n}\) by dummy character tokens which have no meaning, forward it to the model, and measure the reduction of the unit activation value. We repeat this for every candidate concept in the sentences - as a result, we can identify which candidate concept greatly reduce unit activation values. We thus define concepts aligned to each unit as the candidate concept that consistently lower the unit activation across the top \(K\) sentences. More formally, for each unit \(u\), let \(\mathcal{S}=\{s_{1},...,s_{K}\}\) be top \(K\) activated sentences. Since we occlude each candidate concept in sentences, we define the set of candidate concept \(\mathcal{C}=\{c_{1},...,c_{N}\}\), obtained from parsing each sentence in \(\mathcal{S}\). We define the **degree of alignment (DoA)** between a concept \(c\in\mathcal{C}\) and a unit \(u\) as: \[\text{DoA}_{u,c_{n}}=\frac{1}{Z}\sum_{s\in\mathcal{S}}\mathbb{1}(c_{n}\in s)( a_{u}(s)-a_{u}(\text{Occ}_{c_{n}}(s))) \tag{6}\] where \(Z\) is a normalizing factor, and \(a_{u}\) indicates the mean activation of unit \(u\), \(\text{Occ}_{c_{n}}(s)\) is a sentence \(s\) where candidate concept \(c_{n}\) is occluded, and \(\mathbb{1}(c_{n}\in s)\) is an indicator of whether \(c_{n}\) is included in the sentence \(s\). In short, the DoA measures how much a candidate concept contributes to the activation of the unit's top \(K\) sentences. If a candidate concept \(c_{n}\) appears in the topsentences \(\mathcal{S}\) and greatly reduces the activation of unit \(u\), then \(\mathsf{DoA}_{u,c_{n}}\) gets large, implying that the \(c_{n}\) is strongly aligned to unit \(u\).

Since each unit works as a concept detector whose activation value increases with the length of the input sentence (Section 4.2), it is essential to normalize the length of input for fair comparison of DoA values between the concepts that have different lengths one another. Without the length-normalization (_i.e_. each input sentence consists of just one instance of the candidate concept), the DoA metric has a bias to prefer lengthy concepts (_e.g_. phrases) because they typically have more signals that affect the unit activation than short candidate concepts (_e.g_. single words). In this work, we trained a ByteNet for the translation tasks and a VDCNN for the classification tasks, both to analyze properties of representations for language.

In our experiments, we analyze representations of each convolutional block layer.

Although \(M\) is used as a threshold to set how many concepts per unit are considered, different \(M\) values have little influence on quantitative results such as selectivity in Section 4.2. Figure 8 shows the mean and variance of selectivity values with different \(M=[1,3,5,10]\), where there is little variants in the overall trend; the sensitivity of the _replicate_ set is the highest, and that of _one instance_ is runner-up, and that of _random_ is the lowest. Whereas some units are sensitive to specific natural language concepts as shown in Section 4.3, other unites are not sensitive to any concepts at all.

We first define the unit interpretability for unit \(u\) as follows: \[\mathsf{Interpretability}(u)=\begin{cases}1&\text{if max}_{s\in\mathcal{S}}\{a _{u}(s)\}<\text{max}_{i=1,\dots,N}\{r_{i}\}\\ 0&\text{otherwise}\end{cases} \tag{7}\] where \(\mathcal{S}\) is the set of training sentences, \(a_{u}(s)\) is the activation value of unit \(u\), and \(r_{i}\) is the activation value of the sentence that is made up of replicating concept \(c_{i}\). We define unit \(u\) as interpretable when its \(\mathsf{Interpretability}(u)\) equals to 1, and otherwise as non-interpretable. The intuition is that if a replicated sentence that is composed of only one concept has a less activation value than the top-activated sentences, the unit is not sensitive to the concept compared to a sequence of different words.

For example, in unit 001 in the left of Figure 10, we discover that sentence structure involves many _commas_ in top activated sentences.

We introduce some units whose concepts have the shared meaning in Section 4.3. We here refer _concept cluster_ to the concepts that are aligned to the same unit and have similar semantics or grammatical roles. We analyze how clusters are formed in the units and how they vary with the target task and layer depth. Each central heat map represents the number of times each concept pair is aligned to the same unit. Since the concepts in the x, y-axes are ordered by the clustering result, if the diagonal blocks (concept clusters) emerge more strongly, the concepts in the same unit are more likely to have the similar meanings. In Figure 11, the units learned in the classification tasks tend to have stronger concept clusters than those learned in the translation tasks. Particularly, the concept clusters are highly evident in the units learned in DBpedia and AG News dataset. Our intuition is that units might have more benefits to solve the task by clustering similar concepts in the classification than the translation.

We analyze how concept clusters change by layer in each task.

We project each concept to the vector space using the three pretrained embeddings: (1) Glove (Pennington et al., 2014), (2) ConceptNet (Speer et al., 2017), (3) fastText.

Consistent with our intuition, in all datasets, both the document frequency and the delta of expected loss are closely related to the number of units per concept. It concludes that the representations are learned for identifying not only the frequent concepts in the training set and but also the important concepts for solving the target task. In section 4.4, we visualized how concepts are distributed across layers, where the model is trained on AG News dataset and English-to-German Europarl dataset.

In the classification tasks, we expect to find more concepts that are directly related to predicting the output label, as opposed to the translation tasks where the representations may have to include information on most of the words for an accurate translation. While our goal is not to relate each concept to one of the labels, we find several concepts that are more predictive to a particular label than others. Consistent with section 4.4, there are data-specific and task-specific concepts aligned in each layer; _i.e._ {_worst, 2 stars, awful_} at Yelp Review, {_film, ship, school_} at DBpedia, and some key words at translation datasets. Note that Yelp Review and DBpedia is a classification dataset, where the model is required to predict the polarity (_i.e._ +1 or -1) or ontology (_i.e._ Company, Educational Institution, Artist, Athlete, Officeholder, Mean of Transportation, Building, Natural Place, Village, Animal, Plant, Album, Film, Written Work) for given sentence in supervised setting.

Input: Figure 10: Examples of non-interpretable units, their concepts and top 5 activated sentences. Units are from representations learned on english-to-french translation dataset. See section D for details.

Input: Figure 10: Examples of non-interpretable units, their concepts and top 5 activated sentences. Units are from representations learned on english-to-french translation dataset. See section D for details.

Input: Figure 1: We discover the most activated sentences and aligned concepts to the units in hidden representations of deep convolutional networks. Aligned concepts appear frequently in most activated sentences, implying that those units respond selectively to specific natural language concepts.

Input: Figure 14: The Pearson correlations between the number of units per concept and (i) document frequency (top row), (ii) delta of expected loss (bottom row). They are measured at the final layer representation.

Input: Figure 15: 30 concepts selected by the number of aligned units in four encoding layers in VDCNN learned on Yelp Review dataset and DBpedia ontology dataset, and ByteNet learned on the English-to-German, English-to-French, and English-to-Czech parallel corpus. [#] symbol denotes morpheme concept.

Input: Figure 12: Averaged pairwise distances of concepts in each layer per task. Top and bottom row show the concept distances of translation models and classification models, respectively. For projecting concepts into the embedding space, we use three pretrained embedding: Glove, ConceptNet, FastText. See Appendix E.2 for more details.

Input: Figure 13: The number of unique concepts in each layer. It increases with the layer depth, which implies that the units in a deeper layer represent more diverse concepts.

Input: Figure 16: Aligned concepts per each task and their number of occurrences over multiple layers. See Appendix H for more details.

Input: Figure 11: Concept clusters of the last layer representations learned in each task. The more distinct the diagonal blocks are, the stronger the tendency that concepts aligned to the same unit share similar meanings or semantics. See Appendix E for details.

Input: Figure 8: Mean and variance of selectivity values for different M = [1, 3, 5, 10], where M is the number of selected concepts per unit. In all settings, the selectivity of replicate ones is the highest, that of one instance ones is runner-up, and that of random is the lowest near 0.

Input: Figure 7: Correlations between the number of units per concept and (a) document frequency and (b) delta of expected loss. Pearson correlation coefficients are measured at the final layer of the AG News sentence classification task. Some concepts aligned to many units are annotated. Results on other tasks are available in Appendix F.

Input: Figure 11: Concept clusters of the last layer representations learned in each task. The more distinct the diagonal blocks are, the stronger the tendency that concepts aligned to the same unit share similar meanings or semantics. See Appendix E for details.

Input: Figure 4: 30 concepts selected by the number of aligned units in three encoding layers of ByteNet learned on the Europarl translation dataset (left column) and VDCNN learned on AG-News (right column). [#] symbol denotes morpheme concepts. See section 4.4 for details.

Input: Figure 3: Examples of top activated sentences and aligned concepts to some units in several encoding layers of ByteNet and VDCNN. For each unit, concepts and their presence in top K sentences are shown in the same color. [#] symbol denotes morpheme concepts. See section 4.3 for details.

Input: Figure 14: The Pearson correlations between the number of units per concept and (i) document frequency (top row), (ii) delta of expected loss (bottom row). They are measured at the final layer representation.

Input: Figure 15: 30 concepts selected by the number of aligned units in four encoding layers in VDCNN learned on Yelp Review dataset and DBpedia ontology dataset, and ByteNet learned on the English-to-German, English-to-French, and English-to-Czech parallel corpus. [#] symbol denotes morpheme concept.

Input: Figure 12: Averaged pairwise distances of concepts in each layer per task. Top and bottom row show the concept distances of translation models and classification models, respectively. For projecting concepts into the embedding space, we use three pretrained embedding: Glove, ConceptNet, FastText. See Appendix E.2 for more details.

Input: Figure 13: The number of unique concepts in each layer. It increases with the layer depth, which implies that the units in a deeper layer represent more diverse concepts.

Input: Figure 16: Aligned concepts per each task and their number of occurrences over multiple layers. See Appendix H for more details.

Input: Figure 2: Mean and variance of selectivity values over all units in the learned representation for each dataset. Sentences including the concepts that our alignment method discovers always activate units significantly more than random sentences. See section 4.2 for details.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence, the authors' question about "what natural language concepts are captured by units" appears moderately overstated. While the evidence demonstrates that individual units do capture various linguistic patterns (morphemes, words, phrases) and some semantic/syntactic relationships, characterizing all of these as "concepts" may be too broad, as noted in the reviewer's comments. The evidence shows detection of linguistic elements that serve various functions beyond just conceptual/semantic roles. Additionally, the reviewer points out that external validation against semantic resources would help verify the "concept" framing. However, the core finding about units learning interpretable linguistic patterns is well-supported.

<score>3</score>