Weakly Supervised Domain Detection
Yumo Xu Mirella Lapata
cs.CL cs.IR cs.LG




 


Weakly Supervised Domain Detection
    Yumo Xu Mirella Lapata

 Institute for Language, Cognition and Computation

 School of Informatics, University of Edinburgh

 10 Crichton Street, Edinburgh EH8 9AB

 , 
 
    
==============================================================================================================================================================================


  In this paper we introduce domain detection as a new
  natural language processing task. We argue that the ability to
  detect textual segments which are domain-heavy, i.e., sentences or
  phrases which are representative of and provide evidence for a given
  domain could enhance the robustness and portability of various text
  classification applications. We propose an encoder-detector
  framework for domain detection and bootstrap classifiers with
  multiple instance learning (MIL). The model is hierarchically
  organized and suited to multilabel classification.  We demonstrate
  that despite learning with minimal supervision, our model can be
  applied to text spans of different granularities, languages, and
  genres.  We also showcase the potential of domain detection for text
  summarization.





§ INTRODUCTION



Text classification is a fundamental task in Natural Language
processing which has been found useful in a wide spectrum of
applications ranging from search engines enabling users to identify
content on websites, sentiment and social media analysis, customer
relationship management systems, and spam detection. Over the past
several years, text classification has been predominantly modeled as a
supervised learning problem
(e.g., )
for which appropriately labeled data must be collected. Such data is
often domain-dependent (i.e., covering specific topics such as those
relating to “Business” or “Medicine”) and a classifier trained
using data from one domain is likely to perform poorly on another. For
example, the phrase “the mouse died quickly” may indicate
negative sentiment in a customer review describing the hand-held
pointing device or positive sentiment when describing a laboratory
experiment performed on a rodent. The ability to handle a wide
variety of domains[The term “domain” has been permissively
  used in the literature to describe (a) a collection of documents
  related to a particular topic such as user-reviews in Amazon for a
  product category (e.g., books, movies), (b) a type of
  information source (e.g., twitter, news articles), and
  (c) various fields of knowledge (e.g., Medicine, Law,
    Sport). In this paper we adopt the latter definition of domains,
  however, nothing in our approach precludes applying it to different
  domain labels.] has become more pertinent with the rise of
data-hungry machine learning techniques like neural networks and their
application to a plethora of textual media ranging from news articles
to twitter, blog posts, medical journals, Reddit comments, and
parliamentary debates
<cit.>.



The question of how to best deal with multiple domains when training
data is available for one or few of them has met with much interest in
the literature. The field of domain adaptation
<cit.>
aims at improving the learning of a predictive function in a
target domain where there is little or no labeled data, using
knowledge transferred from a source domain where sufficient
labeled data is available. Another line of work
<cit.> assumes that
labeled data may exist for multiple domains, but in insufficient
amounts to train classifiers for one or more of them. The aim of
multi-domain text classification is to leverage all the
available resources in order to improve system performance across
domains simultaneously.






























In this paper we investigate the question of how domain-specific data
might be obtained in order to enable the development of text
classification tools as well as more domain aware applications such as
summarization, question answering, and information extraction.  We
refer to this task as domain detection and assume a fairly
common setting where the domains of a corpus collection are known and
the aim is to identify textual segments which are domain-heavy,
i.e., documents, sentences, or phrases providing evidence for a given
domain.










Domain detection can be formulated as a multilabel
classification problem, where a model is trained to recognize domain
evidence at the sentence-, phrase-, or word-level. By definition then,
domain detection would require training data with fine-grained domain
labels, thereby increasing the annotation burden; we must provide
labels for training domain detectors and for modeling the task
we care about in the first place. In this paper we consider the
problem of fine-grained domain detection from the perspective of
Multiple Instance Learning (MIL; ) and
develop domain models with very little human involvement.  Instead of
learning from individually labeled segments, our model only requires
document-level supervision and optionally prior domain knowledge and
learns to introspectively judge the domain of constituent segments.
Importantly, we do not require document-level domain annotations
either since we obtain these via distant supervision by leveraging
information drawn from Wikipedia.


Our domain detection framework comprises two neural network modules;
an encoder learns representations for words and sentences
together with prior domain information if the latter is available
(e.g., domain definitions), while a detector generates
domain-specific scores for words, sentences, and documents. We obtain
a segment-level domain predictor which is trained end-to-end on
document-level labels using a hierarchical, attention-based neural
architecture <cit.>.  We conduct domain detection
experiments on English and Chinese and measure system performance
using both automatic and human-based evaluation.  Experimental results
show that our model outperforms several strong baselines and is robust
across languages and text genres, despite learning from weak
supervision. We also showcase our model's application potential for
text summarization.




Our contributions in this work are threefold; we propose domain
detection, as a new fine-grained multilabel learning problem which we
argue would benefit the development of domain aware NLP tools; we
introduce a weakly supervised encoder-detector model within
the context of multiple instance learning; and demonstrate that it can
be applied across languages and text genres without modification.




§ RELATED WORK



Our work lies at the intersection of multiple research areas,
including domain adaptation, representation learning,  multiple
instance learning, and topic modeling. We review related work below.



The main difference between HierNet and MilNet: semantic composition and instance composition. 

Semantic composition depicts more abstract high-level
semantics by information compression, but losses information of
individual instances. On the contrary, Instance composition
captures task-specific discriminative features from instances but is
unable to obtain more abstract interactions between instances.

Therefore, semantic composition works better when for a given task the
semantic structure is important but complex, while instance
composition has more advantages when instance distributions are
significant and less dependent on each other. For instance, MilNet treats a document as bags and segments as instances, and
shows superior performance in fine-grained sentiment analysis. One
potential reason is that segments are informative themselves and when
yield document distributions their semantics are not heavily dependent
on each other.




  
Domain Adaptation A variety of domain adaptation methods
<cit.> have been
proposed to deal with the lack of annotated data in novel domains
faced by supervised models.  daume2006domain propose to
learn three separate models, one specific to the source domain, one
specific to the target domain, and a third one representing domain
general information.  A simple yet effective feature augmentation
technique is further introduced in daume2007frustratingly
which finkel2009hierarchical subsequently recast within a
hierarchical Bayesian framework.  More recently,
lu2016general present a general regularization framework for
domain adaptation while


camacho2017babeldomains integrate domain information within
lexical resources. A popular approach within text classification
learns features that are invariant across multiple domains whilst
explicitly modeling the individual characteristics of each domain
<cit.>.








Similar to domain adaptation, our detection task also identifies the
most discriminant features for different domains. However, while
adaptation aims to render models more portable by transferring
knowledge, detection focuses on the domains themselves and identifies
the textual segments which provide the best evidence for their
semantics, allowing to create datasets with explicit domain
labels to which domain adaptation techniques can be further applied.





















Topic Modeling
 Labeled LDA <cit.>. 
 Different from a topic, domain is a high-level concept and its abstractness leads to difficulties in learning the mappings between texts and their labels. For instance, for a piece of restaurant review, we could easily recognize topic clues and categorize it to topic Foods, while detecting the domain  from text would be a trickier problem since (1) the review does not mention  neither directly nor indirectly and (2) the coverage of  is obscure and we are not informed of its relation with Foods. 

For more effective learning, we include two types of domain side information as prior knowledge for domain detection: domain definitions and domain topics.  
Domain descriptions are unstructured sentences, providing general and abstract descriptions of a domain. 
Differently, domain topics are a list of phrases that have an explicit belonging relation with their domains. They are concrete instances giving an approximation of the domain scope. 





  
Multiple Instance Learning Multiple instance learning
(MIL) handles problems where labels are associated with groups or
bags of instances (documents in our case), while instance
labels (segment-level domain labels) are unobserved.  The task is then
to make aggregate instance-level predictions, by inferring labels
either for bags
<cit.> or
jointly for instances and bags
<cit.>.  Our domain
detection model is an example of the latter variant.

Initial MIL models, adopted a relatively strong consistency assumption 
between bag labels and instance labels. For instance, in  binary
classification, a bag was considered positive only if all its instances were
positive
<cit.>.
The assumption was subsequently relaxed by 
investigating prediction combinations
<cit.>.



Within NLP, multiple instance learning has been predominantly applied
to sentiment analysis. kotzias2015group use sentence vectors
obtained by a pre-trained hierarchical CNN <cit.> as
features under a MIL objective which simply averages instance
contributions towards bag classification (i.e., positive/negative
document sentiment).  pappas2014explaining adopt a multiple
instance regression model to assign sentiment scores to specific
product aspects, using a weighted summation of predictions.  More
recently, angelidis2018multiple propose MilNet, a
multiple instance learning network model for sentiment analysis. They
employ an attention mechanism to flexibly weigh predictions and
recognize sentiment-heavy text snippets (i.e., sentences or clauses).

We depart from previous MIL-based work, in devising an encoding module
with self-attention and non-recurrent structure, which is particularly
suitable for modeling long documents efficiently. Compared to MILNet <cit.>, our approach generalizes to
segments of arbitrary granularity; it introduces an instance
scoring function which supports multilabel rather than binary
classification, and takes prior knowledge into account (e.g., domain
definitions) to better inform the model's predictions.



  
Topic Modeling Topic models are built around the idea that
the semantics of a document collection is governed by latent
variables. The aim is therefore to uncover these latent
variables–topics–that shape the meaning of the document
collection. Latent Dirichlet Allocation (LDA;
) is one of the best-known topic models. In
LDA, documents are generated probabilistically using a mixture
over K topics which are in turn characterized by a distribution over
words. And words in a document are generated by repeatedly sampling a
topic according to the topic distribution and selecting a word given
the chosen topic.




Although most topic models are unsupervised, some variants can also
accommodate document-level supervision
<cit.>.  However, these
models are not appropriate for analyzing multiply labeled corpora
since they limit documents to being associated with a single label.

Multi-Multinomial LDA (MM-LDA; ) relaxes
this constraint by modeling each document as a bag of words with a bag
of labels, and topics for each observation are drawn from a shared
topic distribution.


Labeled LDA (L-LDA; ) goes one step further
by directly associating labels with latent topics thereby learning
label-word correspondences.  L-LDA is a natural extension of both LDA
by incorporating supervision and Multinomial Naive Bayes
<cit.> by incorporating a mixture model
<cit.>.

Similar to L-LDA,  is also designed to perform learning and
inference in multi-label settings.  Our model adopts a more general
solution to the credit attribution problem (i.e., the association of
textual units in a document with semantic tags or labels).

Despite learning from a weak and distant signal, our model can produce
domain scores for text spans of varying granularity (e.g., sentences
and phrases) not just words and achieves this with a
hierarchically-organized neural architecture. Aside from learning
through efficient backpropagation, the proposed framework can take
incorporate useful prior information (e.g., pertaining to the labels
and their meaning).



§ PROBLEM FORMULATION




Since each document could belong to multiple domains, we formulate domain detection as a multi-label learning problem. We target at C domains, each of which is associated with a collection of topics 𝒯_c = {𝒯_c^in, 𝒯_c^out}, c ∈ [1, C] comprising of an in-scope topic set and an out-of-scope topic set. 



We formulate domain detection as a multilabel learning problem. 




Our model is trained on samples of document-label pairs. Each document
consists of s sentences x={x_1, …, x_s} and is associated
with  discrete labels y={y^(c)| c ∈[1, C]
}. In this work, domain labels are not annotated manually but
extrapolated from Wikipedia (see Section <ref> for details).
In a non-MIL framework, a model typically learns to predict document
labels by directly conditioning on its sentence representations
h_1, …, h_s or their aggregate.

In contrast, y under MIL is a learned function f_θ of
latent instance-level labels, i.e., . A MIL classifier will therefore first produce domain
scores for all instances (aka sentences), and then learn to integrate
instance scores into a bag (aka document) prediction.


In this paper we further assume that the instance-bag
relation applies to sentences and documents but also to words and
sentences.


In addition, we incorporate prior domain information to facilitate
learning in a weakly supervised setting: each domain is associated
with a definition 𝒰^(c), i.e., a few sentences
providing a high-level description of the domain at hand. For example,
the definition of the “Lifestyle” domain is “the interests,
opinions, behaviors, and behavioral orientations of an individual,
group, or culture”.















Figure <ref> provides an overview of our Domain
Detection Network, which we call .  The
model comprises two modules; an encoder learns
representations for words and sentences whilst incorporating prior
domain information; a detector generates domain scores for
words, sentences, and documents by selectively attending to previously
encoded information. We describe the two modules in more detail below.


We acquire domain embeddings U = [𝐮_1; … ; 𝐮_C] by averaging all the in-scope topic embeddings, i.e., 
𝐮_c = 1 /L^in∑_i ∈ [1, L^in]𝐯_c, i^in.

Therefore, for the cth domain, we have two topic embedding matrices, in which one is for in-scope V_c^in = [𝐯_c, 1^in; …; 𝐯_c, L^in^in] and the other is for out-of-scope  V_c^out = [𝐯_c, 1^out; …; 𝐯_c, L^out^out]. 






§ THE ENCODER MODULE






We learn representations for words and sentences using identical
encoders with separate learning parameters.  Given a document, the two
encoders implement the following steps:

    Z, α= WordEnc (X) 
    
        G = [ g_1; …; g_s]  g = Zα
    H, β= SentEnc (G)


For each sentence X = [ x_1; … ; x_n], the
word-level encoder yields contextualized word representations Z and
their attention weights α.  Sentence embeddings g
are obtained via weighted averaging and then provided as input to the
sentence-level encoder which outputs contextualized representations
H and their attention weights β.

In this work we aim to model fairly long documents (e.g., Wikipedia
articles; see Section <ref> for details). For this reason, our
encoder builds on the Transformer architecture
<cit.>, a recently proposed highly efficient
model which has achieved state-of-the-art performance in machine
translation <cit.> and question answering
<cit.>. The Transformer aims at reducing the fundamental
constraint of sequential computation which underlies most
architectures based on recurrent neural networks. It eliminates
recurrence in favor of applying a self-attention mechanism which
directly models relationships between all words in a sentence,
regardless of their position.







  
Self-Attentive Encoder As shown in Figure <ref>,
the Transformer is a non-recurrent framework comprising m identical
layers. Information on the (relative or absolute) position of each
token in a sequence is represented by the use of positional encodings
which are added to input embeddings (see the bottom of
Figure <ref>). We denote position-augmented inputs in a
sentence with X. Our model uses four layers in both word and
sentence encoders. The first three layers are identical to those in
the Transformer (m=3), comprising a multi-head self-attention
sublayer and a position-wise fully-connected feed-forward network. The
last layer is simply a multi-head self-attention layer yielding
attention weights for subsequent operations.

Single-head attention takes three parameters as input in the
Transformer <cit.>: a query matrix, a key matrix,
and a value matrix. These three matrices are identical and equal to
the inputs X at the first layer of the word encoder. The output
of a single-head attention is calculated via:




    (X,X, X) = (XX^⊺/√(d_x))X.







Multi-head attention allows to jointly attend to information from
different representation subspaces at different positions. This is
done by first applying different linear projections to inputs and then
concatenating them:

    ^(k) = (XW_1^(k),XW_2^(k), XW_3^(k))
    =(^(1),.., ^(r))W_4

where we adopt four heads (r=4) for both word and sentence encoders.
The second sublayer in the Transformer (see Figure <ref>) is
a fully-connected feed-forward network applied to each position
separately and identically:[We omit here the bias term for
  the sake of simplicity.]

    (x) = max(0, xW_5)W_6 .


After sequentially encoding input embeddings through the first three
layers, we obtain contextualized word representations Z∈^d_z × n.  Based on Z, the last multi-head attention
layer in the word encoder yields a set of attention matrices 𝒜 = {A^(k)}_k=1^r for each sentence where A^(k)∈^n × n.

Therefore, when measuring the contributions from words to sentences,
e.g., in terms of domain scores and representations, we can
selectively focus on salient words within the set 𝒜 =
{A^(k)}_k=1^r:

    α = (1/√(n r)∑^r_k ∑^n_ℓA^(k)_⋆, ℓ)

where the  function outputs the salience distribution over words:

    (a_ℓ) = e^a_ℓ/∑_a_ℓ'∈a e^a_ℓ'


and obtain sentence embeddings g = Zα. 

In the same vein, we adopt another self-attentive encoder to obtain
contextualized sentence representations
H∈^d_h × s. The final layer outputs multi-head
attention score matrices ℬ = {B^(k)}_k=1^r
(with B^(k)∈^s × s), and we calculate sentence
salience as:
  
    β = (1/√(sr)∑^r_k ∑^s_j B^(k)_⋆, j).









Prior Information 


In addition to documents (and their domain labels), we might have some
prior knowledge about the domain, e.g., its general semantic content
and the various topics related to it. For example, we might expect
articles from the “Lifestyle” domain to not talk about missiles
or warfare, as these are recurrent themes in the “Military”
domain. As mentioned earlier, throughout this paper we assume we have
domain definitions 𝒰 expressed in a few sentences as prior
knowledge.

Domain definitions share parameters with WordEnc and
SentEnc and are encoded in a definition
matrix U∈^d_h × C. 

Intuitively, identifying the domain of a word might be harder than
that of a sentence; on account of being longer and more expressive,
sentences provide more domain-related cues than words whose meaning
often relies on supporting context.  We thus inject domain
definitions U into our word detector only.




§ THE DETECTOR MODULE





DetNet adopts three detectors corresponding to words,
sentences, and documents:

    P= WordDet (Z, U) 
    
        Q^instc = [q^instc_1; …; q^instc_s]  q^instc = Pα
    Q= SentDet (Q^instc, H) 
    
    
        ỹ = DocDet (Q, β)


WordDet first produces word domain scores using both
lexical semantic information Z and prior (domain)
knowledge U; SentDet yields domain scores for
sentences while integrating downstream instance
signals Q^instc and sentence semantics H; finally,
DocDet makes the final document-level predictions based on
sentence scores.










  
Word Detector Our first detector yields word domain
scores.  For a sentence, we obtain a self-scoring
matrix P^self using its own contextual word semantic
information:

    P^self = tanh (W_z Z).



In contrast to the representations used in
angelidis2018multiple, we generate instance scores from
contextualized representations, i.e., Z.  Since the 
function normally favors single-mode outputs, we adopt tanh(·)
∈ (-1, 1) as our domain scoring function to tailor MIL to our
multilabel scenario.


As mentioned earlier, we employ domain definitions as prior
information at the word level and compute the prior score via:

    P^prior = tanh(max(0, U ^⊺W_u)  Z)





where W_u ∈^d_u × d_z projects prior information
U onto the input semantic space. The prior score matrix
P^prior captures the interactions between domain definitions
and sentential contents.

In this work, we flexibly integrate scoring components with gates, as
shown in Figure <ref>. The key idea is to learn a
prior gate Γ balancing
Equations (<ref>) and (<ref>) via:

    Γ = γσ(W_g, p [Z, P^self, P^prior]) 
    P = Γ⊙P^prior + (J-Γ)  ⊙P^self





where J is an all-ones matrix and P∈^C × n
is the final domain score matrix at the word-level; ⊙ denotes
element-wise multiplication and [·, ·] matrix concatenation.
σ(·) ∈ (0, 1) is the sigmoid function and
Γ∈ (0, γ) the prior gate with scaling
factor γ, a hyperparameter controlling the overall effect of
prior information and instances.[Initially, we expected to
  balance these effects by purely relying on the learned function
  without a scaling factor. This, however, led to poor performance.]
















  
Sentence Detector







The second detector identifies sentences with domain-heavy semantics
based on signals from the sentence encoder, prior information and word
instances. Again we obtain a self-scoring matrix Q^self via:

    Q^self = tanh(W_h H).












After computing sentence scores from sentence-level signals, 

we estimate domain scores from individual words.  We do this by reusing
α in Equation (<ref>), q^instc = Pα. After gathering q^instc for each sentence, we
obtain Q^instc∈^C × s as the full instance score
matrix.


Analogously to the word-level detector (see Equation (<ref>)), we employ  a sentence-level
upward gate Λ to dynamically propagate domain scores from downstream word instances
to sentence bags:

    Λ = λσ(W_ℓ [H, Q^instc, Q^self]) 
    Q =Λ⊙Q^instc +  (J - Λ)  ⊙Q^self

where Q is the final sentence score matrix.






  
Document Detector 
Document-level domain scores are based on the sentence salience
distribution β (see Equation (<ref>)) and are
computed as the weighted average of sentence scores:

    ỹ = Qβ.




We use only document-level supervision for multilabel learning in C
domains. Formally, our training objective is:

    
    ℒ =    min -1/N∑_n^N ∑_c^C 
    [
    y^(n)_clogỹ^(n)_c
       + 
    (1 - y^(n)_c) 
    log (1 -  ỹ^(n)_c)
    )
    ]



    ℒ = 
        min -1/N∑_i^N ∑_c^C 
    log ( 1 + e^-ỹ^(i)_c  y^(i)_c)

where N is the training set size. At test time, we partition domains
into a relevant set and an irrelevant set for unseen samples.  Since
the domain scoring function is tanh(·) ∈ (-1, 1), we use a
threshold of 0 against which ỹ is
calibrated.[If ∀ c ∈ [1, C]: ỹ_c < 0
  holds, we set
  ỹ_c^∗ = 1 and select c^∗ as c^∗ = _cỹ_c to produce a positive prediction.]



§ EXPERIMENTAL SETUP





















  
Datasets DetNet was trained on two datasets
created from
Wikipedia[http://static.wikipedia.org/downloads/2008-06/en]
for English and Chinese.[Available at https://github.com/yumoxu/detnet] Wikipedia articles are organized according
to a hierarchy of categories representing the defining characteristics
of a field of knowledge.  We recursively collect Wikipedia pages by
first determining the root categories based on their match
with the domain name. We then

obtain their subcategories, the subcategories of these
subcategories, and so on. We treat all pages associated with a category
as representative of the domain of its root category.  

In our experiments we used seven target domains: “Business and
  Commerce” (), “Government and Politics”
(), “Physical and Mental Health” (),
“Law and Order” (), “Lifestyle” (),
“Military” (), and “General Purpose”
().  Exceptionally,  does not have a natural
root category. We leverage Wikipedia's 12 Main
Categories[https://en.wikipedia.org/wiki/Portal:Contents/Categories]
to ensure that  is genuinely different from the other six
domains. We used 5,000 pages for each domain.
Table <ref> shows various statistics on our dataset. 








  
System Comparisons
We constructed three variants of  to explore the contribution
of different model components.   has a single-level hierarchical
structure, treating only sentences as instances and documents as bags;
while   has a two-level hierarchical structure (the instance-bag
relation applies to words-sentences and sentences-documents); finally,
 is our full model which is fully hierarchical and equipped
with prior information (i.e., domain definitions).  We also compared
 to a variety of related systems which include:

Major: the  Majority domain label applies  to all instances. 

L-LDA: Labeled LDA <cit.> is a topic model
that constrains LDA by defining a one-to-one correspondence between
LDA's latent topics and observed labels. This allows L-LDA to directly
learn word-label correspondences. We obtain domain scores for words
through the topic-word-count matrix M∈^C × V which
is computed during training:

    M̃ = M^⊺ + β/∑^C_c M^⊺_*,c + C * β

where C and V are the number of domain labels and the size of
vocabulary, respectively. Scalar β is a prior value set to 1/C
and matrix M̃∈^V × C consists of word
scores over domains. Following the snippet extraction approach
proposed in ramage2009labeled, L-LDA can also be used to
score sentences as the expected probability that the domain label had
generated each word. For more details on L-LDA, we refer the interested
reader to ramage2009labeled.








HierNet: A hierarchical neural network model described
in angelidis2018multiple which produces document-level
predictions by attentively integrating sentence representations. For
this model we used word and sentence encoders identical to
DetNet. HierNet does not generate instance-level
predictions, however, we assume that document-level predictions apply
to all sentences. 



MilNet: 
A variant of the
MIL-based model introduced in <cit.> which considers
sentences as instances and documents as bags (while DetNet
generalizes the instance-bag relationship to words and sentences).  To
make  comparable to our system, we use an encoder identical to , i.e., two Transformer encoders for words and sentences, respectively. Thus,  differs from  in two respects: (a) word representations are simply averaged without
word-level attention to build sentence embeddings and (b) context-free
sentence embeddings generate sentence domain scores before being fed
to the sentence encoder.









  
Implementation Details We used 16 shuffled samples in a
batch where the maximum document length was set to 100 sentences with the excess clipped. Word embeddings were initialized randomly with 256
dimensions. All weight matrices in the model were initialized with the
fan-in trick <cit.> and biases were initialized
with zero.  Apart from using layer normalization <cit.> in
the encoders, we applied batch normalization <cit.> and a
dropout rate of 0.1 in the detectors to accelerate model training.
We trained the model with the Adam optimizer <cit.>. 

We set all three gate
scaling factors in our model to 0.1.  Hyper-parameters were optimized
on the development set.  To make our experiments easy to replicate, we
release our PyTorch <cit.> source
code.[Available at https://github.com/yumoxu/detnet]



§ AUTOMATIC EVALUATION




In this section we present the results of our automatic evaluation for
sentence and document predictions. Problematically, for sentence
predictions we do not have gold-standard domain labels (we have only
extrapolated these from Wikipedia for documents). We therefore
developed an automatic approach for creating silver standard domain
labels which we describe below. 



  
Test Data Generation In order to obtain sentences with
domain labels, we exploit lead sentences in Wikipedia
articles. Lead sentences typically define the article's subject matter
and emphasize its topics of
interest.[https://en.wikipedia.org/wiki/Lead_paragraph] As
most lead sentences contain domain-specific content we can fairly
confidently assume that document-level domain labels will apply.  To
validate this assumption, we randomly sampled 20 documents
containing 220 lead sentences and asked two annotators to label these
with domain labels. Annotators overwhelmingly agreed in their
assignments with the document labels, the (average) agreement was
K=0.89 using Cohen’s Kappa coefficient.





We used the lead sentences to create pseudo documents simulating real
documents whose sentences cover multiple domains. To ensure sentence
labels are combined reasonably (e.g.,  is not likely to
coexist with ), prior to generating synthetic documents,
we traverse the training set and acquire all domain combinations
𝒮,

e.g., 𝒮 = {{},{,
}}. We then gather lead sentences representing the same domain
combinations. We generate synthetic documents with a maximum length of 100
sentences (we also clip real documents to the same length). 

Algorithm 1 shows the pseudocode for document generation.  We first
sample document labels, then derive candidate label sets for sentences
by introducing  and a noisy label ϵ. 
After sampling sentences for each
domain, we shuffle them to achieve domain-varied sentence contexts.
We created two synthetic datasets for English and Chinese.  Detailed
statistics are shown in Table <ref>.




  
Evaluation Metric We evaluate system performance
automatically using label-based  <cit.>, a
widely-used metric for multilabel classification.  It measures model
performance for each label specifically and then macro-averages
the results. For each class, given a confusion matrix
([  ;   ])
containing the number of samples classified as true positive, false
positive, true negative, and false negative,  is calculated as
1/C∑_c=1^C2_c/2_c+_c+_c
where C is the number of domain labels. 







  
Results Our results are summarized in
Table <ref>.  

We first report domain detection results for documents, since reliable
performance on this task is a prerequisite for more fine-grained
domain detection.  As shown in Table <ref>,
DetNet does well on document-level domain detection,
managing to outperform systems over which it has no clear advantage
(such as HierNet or MilNet). 

















As far as sentence-level prediction is concerned, all DetNet
variants significantly outperform all comparison systems. Overall,
 is the best system achieving 54.37% and 55.88%  on
English (en) and Chinese (zh), respectively. It outperforms  by 17.25% on English and 11.51% on Chinese.  The
performance of the fully hierarchical model  is better than ,
showing positive effects of directly incorporating word-level domain
signals.

We also observe that prior information is generally helpful on both
languages and both tasks.












§ HUMAN EVALUATION
 





 


Aside from automatic evaluation, we also assessed model performance
against human elicited domain labels for sentences and words.  The
purpose of this experiment was threefold: (a) to validate the results
obtained from automatic evaluation; (b) to evaluate finer-grained
model performance at the word level; and (c) to examine whether our
model generalizes to non-Wikipedia articles. For this, we created a
third test set from the New York
Times[https://catalog.ldc.upenn.edu/LDC2008T19], in addition
to our Wikipedia-based English and Chinese datasets. For all three
corpora, we randomly sampled two documents for each domain, and then
from each document, we sampled one long paragraph or a few consecutive
short paragraphs containing  sentences.  Amazon Mechanical
Turkers were asked to read these sentences and assign a domain based
on the seven labels used in this paper (multiple labels were
allowed). Participants were provided with domain definitions. We
obtained five annotations per sentence and adopted the majority label
as the sentence's domain label. We obtained two annotated datasets for
English (Wiki-en and NYT-en) and one for Chinese (),
consisting of 122/14, 111/11, and 117/12 sentences/documents each.

Word-level domain evaluation is more challenging; taken
out-of-context, individual words might be uninformative or carry
meanings compatible with multiple domains. Expecting crowdworkers to
annotate domain labels word-by-word with high confidence, might be
therefore problematic. In order to reduce annotation complexity, we
opted for a retrieval-style task for word evaluation. Specifically,
AMT workers were given a sentence and its domain label (obtained from
the sentence-level elicitation study described above), and asked to
highlight which words they considered consistent with the domain of
the sentence. We used the same corpora/sentences as in our first AMT
study.  Analogously, words in each sentence were annotated by five
participants and their labels were determined by majority agreement.


Fully hierarchical variants of our model (i.e., , ) and L-LDA are
able to produce word-level predictions; we thus retrieved the words
within a sentence whose domain score was above the threshold of 0 and
compared them against the labels provided by
crowdworkers. MilNet and  can only make sentence-level
predictions. In this case, we assume that the sentence domain applies
to all words therein. HierNet can
only produce document-level predictions based on which we generate
sentence labels and further assume that these apply to sentence words
too. 

For L-LDA, we can produce word-level scores through the topic-word-count matrix M∈^C × V computed during model training:

    M̃ = M^⊺ + β/∑^C_c M^⊺_*,c + C * β

where C and V are the number of domain labels and the size of vocabulary, respectively. Scalar β is a prior value set to 1/C and matrix M̃∈^V × C consists of word scores over domains. For more details on learning and inference for L-LDA, we refer the interested reader to ramage2009labeled.

Again, we report  which we compute as 2p^∗
  r^∗/p^∗+r^∗ where precision p^∗ and recall r^∗
are both averaged over all words.



























We show model performance against AMT domain labels in
Table <ref>. Consistent with the automatic evaluation
results, DetNet variants are the best performing models on
the sentence-level task. On the Wikipedia datasets,  or
 outperform all baselines and  by a large margin, showing
that word-level signals can indeed help detect sentence domains.
Although statistical models are typically less accurate when they are
applied to data that has a different distribution from the training
data,  works surprisingly well on NYT, substantially
outperforming all other systems. We also notice that prior information
is useful in making domain predictions for NYT sentences: since our
models are trained on Wikipedia, prior domain definitions largely
alleviate the genre shift to non-Wikipedia sentences.
Table <ref> provides a breakdown of the
performance of  across domains. Overall, the model performs
worst on  and  domains (which are very broad)
and best on  and  (which are very narrow).

With regard to word-level evaluation,  and  are the
best systems and are significantly better against all comparison
models by a wide margin, except L-LDA.  The latter is a strong domain
detection system at the word-level since it is able to directly
associate words with domain labels (see
Equation (<ref>)) without resorting to document- or
sentence-level predictions. However, our two-level hierarchical model
is superior considering all-around performance across sentences and
documents.  The results here accord with our intuition from previous
experiments: hierarchical models outperform simpler variants
(including MilNet) since they are able to capture and exploit
fine-grained domain signals relatively accurately. Interestingly,
prior information does not seem to have an effect on the Wikipedia
datasets, but is useful when transferring to NYT.  We also observe
that models trained on the Chinese datasets perform consistently
better than English. Analysis of the annotations provided by
crowdworkers revealed that the ratio of domain words in Chinese is
higher compared to English (27.47% vs. 13.86% in Wikipedia and
16.42% in NYT), possibly rendering word retrieval in Chinese an
easier task.








Table <ref> shows the 15 most representative domain words
identified by our model () on Wiki-en for our seven
domains. We obtained this list by weighting word domain scores P with
their attention scores:

    P^∗ = P⊙ [α; …; α]^⊺
 
and ranking all words in the development set according to
P^∗, separately for each domain. Since words appearing in
different contexts are usually associated with multiple domains, we
determine a word's ranking for a given domain based on the highest
score.  As shown in Table <ref>, biosecurity and
authoritarianism are prevalent in both  and
 domains. Interestingly, with contextualized word
representations, fairly general English words are recognized as domain
heavy. For example, technique is a strong domain word in
 and 420 in  (the latter is slang for
the consumption of cannabis and highly associated with government
regulations).

For comparison, we also show the top domain words identified by L-LDA
via matrix M̃ (see
Equation (<ref>)). To produce meaningful output, we
have removed stop words and punctuation tokens, which are given very
high domain scores by  (this is not entirely surprising
since M̃ is based on simple co-occurrence). Notice that
no such post-processing is necessary for our model.  As shown in
Table <ref>, the top domain words identified by L-LDA (on the
right) are more general and less informative, compared to those from
 (on the left).










§ DOMAIN-SPECIFIC SUMMARIZATION




In this section we illustrate how fine-grained domain scores can be
used to produce domain summaries, following an extractive,
unsupervised approach. We assume the user specifies the domains they
are interested in a priori (e.g., , ) and the
system returns summaries targeting the semantics of these domains.

Specifically, we introduce , an extension of the well-known
 algorithm <cit.>, which incorporates
domain signals acquired by .  For each document,
 builds a directed graph G=(V, E) with nodes V
corresponding to sentences, and undirected edges E whose weights are
computed based on sentence similarity.  Specifically, edge weights are
represented with matrix E where each element E_i, j
corresponds to the transition probability from vertex i to vertex
j.  Following barrios2016variations, E_i, j is
computed with the Okapi BM25 algorithm <cit.>, a
probabilistic version of TF-IDF, and small weights (<0.001) are set
to zeros.  Unreachable nodes are further pruned to acquire the final
vertex set V.














To enhance  with domain information, we first multiply
sentence-level domain scores Q with their corresponding attention
scores:

    Q^∗ = Q⊙ [β; …; β]^⊺.
 
and for a given domain c, we can extract a (domain) sentence score
vector q^∗ = Q^∗_c, *∈^1×
  s. Then, from q^∗, we produce vector q̃∈^1× |V| representing a distribution of domain signals
over sentences:

    q̂ = [q^∗_i]_i ∈ V
    q̃ = (q̂ - q̂_min/q̂_max - q̂_min)



In order to render domain signals in different sentences more
discernible,

we scale all elements in q̂ to [0, 1] before obtaining a
legitimate distribution with the  function.  Finally, we
integrate the domain component into the original transition matrix as:

    Ẽ = ϕ * q̂ + (1-ϕ) * E
 
where ϕ∈ (0, 1) controls the extent to which
domain-specific information influences sentence selection for the
summarization task; higher ϕ will lead to summaries which are
more domain-relevant.  Here, we empirically set ϕ=0.3.  The main
difference between  and  is that  treats 1-
ϕ as a damping factor and a uniform probability distribution is
applied to q̂.

In order to decide which sentence to include in the summary, a node’s
centrality is measured using a graph-based ranking algorithm
<cit.>. Specifically, we run a Markov chain
with Ẽ on G until it converges to the stationary
distribution e^∗ where each element denotes the salience of
a sentence.  In the proposed  algorithm, e^∗
jointly expresses the importance of a sentence in the document
and its relevance to the given domain (controlled by ϕ).

We rank sentences according to e^∗ and select the top K
ones, subject to a budget (e.g., 100 words).

We ran a judgment elicitation study on summaries produced by
 and .  Participants were provided with domain
definitions and asked to decide which summary was best according to
the criteria of: Informativeness (does the summary contain
more information about a specific domain, e.g., “Government and
Politics”?), Succinctness (does the summary avoid
unnecessary detail and redundant information?), and Coherence
(does the summary make logical sense?). Amazon Mechanical Turk (AMT)
workers were allowed to answer “Both” or “Neither” in cases where
they could not discriminate between summaries.  We sampled 50 summary
pairs from the English Wikipedia development set. We collected three
responses per summary pair and determined which system participants
preferred based on majority agreement.




Table <ref> shows the proportion of times AMT workers
preferred each system according to the criteria of Informativeness,
Succinctness, Coherence, and overall. As can be seen, participants
find  summaries more informative and coherent.  While it is
perhaps not surprising for  to produce summaries which are
domain informative since it explicitly takes domain signals into
account, it is interesting to note that focusing on a specific domain
also helps discard irrelevant information and produce more coherent
summaries.  This, on the other hand, possibly renders 's
summaries more verbose (see the Succinctness ratings in
Table <ref>).


Figure <ref> shows example summaries for the Wikipedia
article Arms Industry for domains  and
.[https://en.wikipedia.org/wiki/Arms_industry]
Both summaries begin with a sentence which introduces the arms
industry to the reader. When  is the domain of interest,
the summary focuses on military products such as guns and missiles.
When the domain changes to , the summary puts more
emphasis on trade, e.g., market competition and companies doing
military business, such as Boeing and Eurofighter.


















§ CONCLUSIONS



In this work, we proposed an encoder-detector framework for domain
detection. Leveraging only weak domain supervision, our model achieves
results superior to competitive baselines across different languages,
segment granularities, and text genres. Aside from identifying domain
specific training data, we also show that our model holds promise for
other natural language tasks, such as text summarization.  Beyond
domain detection, we hope that some of the work described here might
be of relevance to other multilabel classification problems such as
sentiment analysis <cit.>, relation extraction
<cit.>, and named entity recognition
<cit.>. More generally, our experiments show
that the proposed framework can be applied to textual data using
minimal supervision, significantly alleviating the annotation
bottleneck for text classification problems. 

A key feature in achieving performance superior to competitive
baselines is the hierarchical nature of our model, where
representations are encoded step-by-step, first for words, then for
sentences, and finally for documents. The framework flexibly
integrates prior information which can be used to enhance the
otherwise weak supervision signal or to render the model more robust
across genres. In the future, we would like to investigate
semi-supervised instantiations of MIL, where aside from bag labels,
small amounts of instance labels are also available
<cit.>.  It would also be interesting to examine how
the label space influences model performance, especially since in our
scenario the labels are extrapolated from Wikipedia and might be naturally
noisy and/or ambiguous.  




§ ACKNOWLEDGMENTS

The authors would like to thank the anonymous reviewers and the action
editor, Yusuke Miyao, for their valuable feedback. We acknowledge the
financial support of the European Research Council (Lapata; award
number 681760). This research is based upon work supported in part by
the Office of the Director of National Intelligence (ODNI),
Intelligence Advanced Research Projects Activity (IARPA), via contract
FA8650-17-C-9118.  The views and conclusions contained herein are
those of the authors and should not be interpreted as necessarily
representing the official policies or endorsements, either expressed
or implied, of the ODNI, IARPA, or the U.S. Government. The
U.S. Government is authorized to reproduce and distribute reprints for
Governmental purposes notwithstanding any copyright annotation
therein.


51
natexlab#1#1

[Andrews and Hofmann(2004)]andrews2004multiple
Stuart Andrews and Thomas Hofmann. 2004.
Multiple instance learning via disjunctive programming boosting.
In Advances in Neural Information processing Systems, pages
  65–72. Curran Associates, Inc.

[Angelidis and Lapata(2018)]angelidis2018multiple
Stefanos Angelidis and Mirella Lapata. 2018.
Multiple instance learning networks for fine-grained sentiment
  analysis.
Transactions of the Association for Computational Linguistics,
  6:17–31.

[Arnold et al.(2007)Arnold, Nallapati, and
  Cohen]arnold2007comparative
Andrew Arnold, Ramesh Nallapati, and William W Cohen. 2007.
A comparative study of methods for transductive transfer learning.
In Proceedings of the 2007 IEEE International Conference on
  Data Mining, pages 77–82, Omaha, NE, USA.

[Ba et al.(2016)Ba, Kiros, and Hinton]ba2016layer
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016.
Layer normalization.
arXiv preprint arXiv:1607.06450.

[Barrios et al.(2016)Barrios, López, Argerich, and
  Wachenchauzer]barrios2016variations
Federico Barrios, Federico López, Luis Argerich, and Rosa Wachenchauzer.
  2016.
Variations of the similarity function of textrank for automated
  summarization.
arXiv preprint arXiv:1602.03606.

[Blei et al.(2003)Blei, Ng, and Jordan]blei2003latent
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.
Latent dirichlet allocation.
Journal of machine Learning research, 3(Jan):993–1022.

[Blitzer et al.(2006)Blitzer, McDonald, and
  Pereira]blitzer2006domain
John Blitzer, Ryan McDonald, and Fernando Pereira. 2006.
Domain adaptation with structural correspondence learning.
In Proceedings of the 2006 Conference on Empirical Methods in
  Natural Language Processing, pages 120–128, Sydney, Australia.

[Bousmalis et al.(2016)Bousmalis, Trigeorgis, Silmerman, Krishnan, and
  Erhan]Bousmalis:ea:2016
Konstantinos Bousmalis, George Trigeorgis, Nathan Silmerman, Dilip Krishnan,
  and Dumitru Erhan. 2016.
Domain separation networks.
In Advances in Neural Information Processing Systems, 29,
  pages 343–351.

[Camacho-Collados and Navigli(2017)]camacho2017babeldomains
Jose Camacho-Collados and Roberto Navigli. 2017.
Babeldomains: Large-scale domain labeling of lexical resources.
In Proceedings of the 15th Conference of the European Chapter
  of the Association for Computational Linguistics, volume 2, pages 223–228,
  Valencia, Spain.

[Carbonetto et al.(2008)Carbonetto, Dorkó, Schmid, Kück, and
  De Freitas]carbonetto2008learning
Peter Carbonetto, Gyuri Dorkó, Cordelia Schmid, Hendrik Kück, and Nando
  De Freitas. 2008.
Learning to recognize objects with little supervision.
International Journal of Computer Vision, 77(1-3):219–237.

[Chen and Cardie(2018)]Chen:Cardie:2018
Xilun Chen and Claire Cardie. 2018.
Multinomial adversarial networks for multi-domain text
  classification.
In Proceedings of the 2018 Conference of the North American
  Chapter of the Association for Computational Linguistics: Human Language
  Technologies, pages 1226–1240, New Orleans, Louisiana.

[Conneau et al.(2017)Conneau, Schwenk, Barrault, and
  Lecun]Conneau:ea:2017
Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann Lecun. 2017.
Very deep convolutional networks for text classification.
In Proceedings of the 15th Conference of the European Chapter
  of the Association for Computational Linguistics, pages 1107–1116,
  Valencia, Spain.

[Daume III(2007)]daume2007frustratingly
Hal Daume III. 2007.
Frustratingly easy domain adaptation.
In Proceedings of the 45th Annual Meeting of the Association of
  Computational Linguistics, pages 256–263, Prague, Czech Republic.

[Daume III and Marcu(2006)]daume2006domain
Hal Daume III and Daniel Marcu. 2006.
Domain adaptation for statistical classifiers.
Journal of Artificial Intelligence Research, 26:101–126.

[Denil et al.(2014)Denil, Demiraj, Kalchbrenner, Blunsom, and
  de Freitas]Denil:ea:2014
Misha Denil, Alban Demiraj, Nal Kalchbrenner, Phil Blunsom, and Nando
  de Freitas. 2014.
Modelling, visualising and summarising documents with a single
  convolutional neural network.
Technical report, University of Oxford.

[Dietterich et al.(1997)Dietterich, Lathrop, and
  Lozano-Pérez]dietterich1997solving
Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez. 1997.
Solving the multiple instance problem with axis-parallel rectangles.
Artificial Intelligence, 89(1-2):31–71.

[Finkel and Manning(2009)]finkel2009hierarchical
Jenny Rose Finkel and Christopher D Manning. 2009.
Hierarchical bayesian domain adaptation.
In Proceedings of Human Language Technologies: The 2009 Annual
  Conference of the North American Chapter of the Association for Computational
  Linguistics, pages 602–610, Boulder, Colorado.

[Glorot and Bengio(2010)]glorot2010understanding
Xavier Glorot and Yoshua Bengio. 2010.
Understanding the difficulty of training deep feedforward neural
  networks.
In Proceedings of the 13th International Conference on
  Artificial Intelligence and Statistics, pages 249–256, Sardinia, Italy.

[Ioffe and Szegedy(2015)]ioffe2015batch
Sergey Ioffe and Christian Szegedy. 2015.
Batch normalization: Accelerating deep network training by reducing
  internal covariate shift.
In Proceedings of the 32nd International Conference on Machine
  Learning, pages 448–456, Lille, France.

[Iyyer et al.(2015)Iyyer, Manjunatha, Boyd-Graber, and
  Daumé III]iyyer-EtAl:2015:ACL-IJCNLP
Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015.
Deep unordered composition rivals syntactic methods for text
  classification.
In Proceedings of the 53rd Annual Meeting of the Association
  for Computational Linguistics and the 7th International Joint Conference on
  Natural Language Processing, pages 1681–1691, Beijing, China.

[Jiang and Zhai(2007)]jiang2007instance
Jing Jiang and ChengXiang Zhai. 2007.
Instance weighting for domain adaptation in nlp.
In Proceedings of the 45th Annual Meeting of the Association of
  Computational Linguistics, pages 264–271, Prague, Czech Republic.

[Keeler and Rumelhart(1992)]keeler1992self
Jim Keeler and David E Rumelhart. 1992.
A self-organizing integrated segmentation and recognition neural net.
In Advances in Neural Information Processing Systems, pages
  496–503. Morgan-Kaufmann.

[Kim(2014)]kim:2014:EMNLP2014
Yoon Kim. 2014.
Convolutional neural networks for sentence classification.
In Proceedings of the 2014 Conference on Empirical Methods in
  Natural Language Processing, pages 1746–1751, Doha, Qatar.

[Kingma and Ba(2014)]KingmaB14
Diederik P. Kingma and Jimmy Ba. 2014.
Adam: A method for stochastic optimization.
CoRR, abs/1412.6980.

[Kotzias et al.(2015)Kotzias, Denil, De Freitas, and
  Smyth]kotzias2015group
Dimitrios Kotzias, Misha Denil, Nando De Freitas, and Padhraic Smyth. 2015.
From group to individual labels using deep features.
In Proceedings of the 21th ACM SIGKDD International Conference
  on Knowledge Discovery and Data Mining, pages 597–606, New York, NY, USA.

[Lacoste-Julien et al.(2009)Lacoste-Julien, Sha, and
  Jordan]lacoste2009disclda
Simon Lacoste-Julien, Fei Sha, and Michael I Jordan. 2009.
Disclda: Discriminative learning for dimensionality reduction and
  classification.
In Advances in Neural Information Processing Systems, pages
  897–904. Curran Associates, Inc.

[Li and Zong(2008)]Li:Zong:2008
Shoushan Li and Chengqing Zong. 2008.
Multi-domain sentiment classification.
In Proceedings of the 46th Annual Meeting of the Association
  for Computational Linguistics: Human Language Technologies, pages 257–260,
  Columbus, Ohio.

[Lu et al.(2016)Lu, Chieu, and Löfgren]lu2016general
Wei Lu, Hai Leong Chieu, and Jonathan Löfgren. 2016.
A general regularization framework for domain adaptation.
In Proceedings of the 2016 Conference on Empirical Methods in
  Natural Language Processing, pages 950–954, Austin, Texas, USA.

[Maron and Ratan(1998)]maron1998multiple
Oded Maron and Aparna Lakshmi Ratan. 1998.
Multiple-instance learning for natural scene classification.
In Proceedings of the Annual International Conference on
  Machine Learning, volume 98, pages 341–349.

[Mcauliffe and Blei(2008)]mcauliffe2008supervised
Jon D Mcauliffe and David M Blei. 2008.
Supervised topic models.
In Advances in Neural Information Processing Systems, pages
  121–128. Curran Associates, Inc.

[McCallum and Nigam(1998)]McCallum:Nigam:1998
Andrew McCallum and Kamal Nigam. 1998.
A comparison of event models for naive Bayes text classification.
In Proceeedings of the AAAI-98 Workshop on Learning For Text
  Categorization, volume 752, pages 41–48, Madison, WI, USA.

[Mihalcea and Tarau(2004)]mihalcea2004textrank
Rada Mihalcea and Paul Tarau. 2004.
Textrank: Bringing order into text.
In Proceedings of the 2004 conference on empirical methods in
  natural language processing, pages 404–411, Barcelona, Spain.

[Noreen(1989)]noreen1989computer
Eric W Noreen. 1989.
Computer-intensive methods for testing hypotheses.
Wiley New York.

[Pan et al.(2010)Pan, Yang et al.]pan2010survey
Sinno Jialin Pan, Qiang Yang, et al. 2010.
A survey on transfer learning.
IEEE Transactions on Knowledge and Data Engineering,
  22(10):1345–1359.

[Pappas and Popescu-Belis(2014)]pappas2014explaining
Nikolaos Pappas and Andrei Popescu-Belis. 2014.
Explaining the stars: Weighted multiple-instance learning for
  aspect-based sentiment analysis.
In Proceedings of the 2014 Conference on Empirical Methods in
  Natural Language Processing, pages 455–466, Doha, Qatar.

[Paszke et al.(2017)Paszke, Gross, Chintala, Chanan, Yang, DeVito,
  Lin, Desmaison, Antiga, and Lerer]paszke2017automatic
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary
  DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017.
Automatic differentiation in pytorch.
In Advances in Neural Information Processing Systems. Curran
  Associates, Inc.

[Ramage et al.(2009a)Ramage, Hall, Nallapati, and
  Manning]ramage2009labeled
Daniel Ramage, David Hall, Ramesh Nallapati, and Christopher D Manning.
  2009a.
Labeled lda: A supervised topic model for credit attribution in
  multi-labeled corpora.
In Proceedings of the 2009 Conference on Empirical Methods in
  Natural Language Processing, pages 248–256, Suntec, Singapore.

[Ramage et al.(2009b)Ramage, Heymann, Manning, and
  Garcia-Molina]ramage2009clustering
Daniel Ramage, Paul Heymann, Christopher D Manning, and Hector Garcia-Molina.
  2009b.
Clustering the tagged web.
In Proceedings of the Second ACM International Conference on
  Web Search and Data Mining, pages 54–63, Barcelona, Spain.

[Robertson et al.(1995)Robertson, Walker, Jones, Hancock-Beaulieu,
  Gatford et al.]robertson1995okapi
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu,
  Mike Gatford, et al. 1995.
Okapi at trec-3.
Nist Special Publication Sp, 109:109.

[Surdeanu et al.(2012)Surdeanu, Tibshirani, Nallapati, and
  Manning]surdeanu-EtAl:2012:EMNLP-CoNLL
Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D. Manning.
  2012.
Multi-instance multi-label learning for relation extraction.
In Proceedings of the 2012 Joint Conference on Empirical
  Methods in Natural Language Processing and Computational Natural Language
  Learning, pages 455–465, Jeju Island, Korea.

[Tang et al.(2017)Tang, Zhang, Zhang, Wu, and
  Zhuang]tang-EtAl:2017:EMNLP2017
Siliang Tang, Ning Zhang, Jinjiang Zhang, Fei Wu, and Yueting Zhuang. 2017.
Nite: A neural inductive teaching framework for domain specific ner.
In Proceedings of the 2017 Conference on Empirical Methods in
  Natural Language Processing, pages 2652–2657, Copenhagen, Denmark.

[Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones,
  Gomez, Kaiser, and Polosukhin]vaswani2017attention
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
  Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.
Attention is all you need.
In Advances in Neural Information Processing Systems, pages
  6000–6010. Curran Associates, Inc.

[Wei et al.(2014)Wei, Wu, and Zhou]wei2014scalable
Xiu-Shen Wei, Jianxin Wu, and Zhi-Hua Zhou. 2014.
Scalable multi-instance learning.
In Proceedings of the 2014 IEEE International Conference on
  Data Mining, pages 1037–1042, Shenzhen, China.

[Weidmann et al.(2003)Weidmann, Frank, and
  Pfahringer]weidmann2003two
Nils Weidmann, Eibe Frank, and Bernhard Pfahringer. 2003.
A two-level learning method for generalized multi-instance problems.
In Proceedings of the European Conference on Machine Learning,
  pages 468–479, Warsaw, Poland.

[Wu and Huang(2015)]Wu:Huang:2015
Fangzhao Wu and Yongfeng Huang. 2015.
Collaborative multi-domain sentiment classification.
In Proceedings of 2015 International Conference on Data
  Mining, pages 459–468, Atlanic City, New Jersey, USA.

[Yang et al.(2016)Yang, Yang, Dyer, He, Smola, and
  Hovy]Zichao:ea:2016
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy.
  2016.
Hierarchical attention networks for document classification.
In Proceedings of the 2016 Conference of the North American
  Chapter of the Association for Computational Linguistics: Human Language
  Technologies, pages 1480–1489, San Diego, California.

[Yu et al.(2018)Yu, Dohan, Luong, Zhao, Chen, Norouzi, and
  Le]yu2018qanet
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad
  Norouzi, and Quoc V Le. 2018.
Qanet: Combining local convolution with global self-attention for
  reading comprehension.
arXiv preprint arXiv:1804.09541.

[Zhang and Zhou(2014)]zhang2014review
Min-Ling Zhang and Zhi-Hua Zhou. 2014.
A review on multi-label learning algorithms.
IEEE Transactions on Knowledge and Data Engineering,
  26(8):1819–1837.

[Zhang et al.(2002)Zhang, Goldman, Yu, and Fritts]zhang2002content
Qi Zhang, Sally A Goldman, Wei Yu, and Jason E Fritts. 2002.
Content-based image retrieval using multiple-instance learning.
In Proceedings of the 19th International Conference on Machine
  Learning, volume 2, pages 682–689, Sydney, Australia.

[Zhang et al.(2016)Zhang, Marshall, and Wallace]Ye:et:al:2016
Ye Zhang, Iain Marshall, and Byron C. Wallace. 2016.
Rationale-augmented convolutional neural networks for text
  classification.
In Proceedings of the 2016 Conference on Empirical Methods in
  Natural Language Processing, pages 795–804, Austin, Texas.

[Zhou et al.(2009)Zhou, Sun, and Li]zhou2009multi
Zhi-Hua Zhou, Yu-Yin Sun, and Yu-Feng Li. 2009.
Multi-instance learning by treating instances as non-iid samples.
In Proceedings of the 26th Annual International Conference on
  Machine Learning, pages 1249–1256, Montreal, Quebec, Canada.


acl_natbib

