<p>Firstly, see <a href="https://stackoverflow.com/questions/34090734/how-to-use-nltk-regex-pattern-to-extract-a-specific-phrase-chunk">How to use nltk regex pattern to extract a specific phrase chunk?</a> </p>

<p>Lets see what's the POS tags for the sentence:</p>

<pre><code>from nltk import word_tokenize, pos_tag

text = "Operating profit margin was 8.3%, compared to 11.8% a year earlier."
pos_tag(word_tokenize(text))
</code></pre>

<p>[out]:</p>

<pre><code>[('Operating', 'NN'),
 ('profit', 'NN'),
 ('margin', 'NN'),
 ('was', 'VBD'),
 ('8.3', 'CD'),
 ('%', 'NN'),
 (',', ','),
 ('compared', 'VBN'),
 ('to', 'TO'),
 ('11.8', 'CD'),
 ('%', 'NN'),
 ('a', 'DT'),
 ('year', 'NN'),
 ('earlier', 'RBR'),
 ('.', '.')]
</code></pre>

<h1>First gotcha! No <code>JJ</code> in any of the tags</h1>

<p>There's no <code>JJ</code> tag in any of the POS in that sentence. </p>

<h1>Lets head back to the paper <a href="https://arxiv.org/pdf/1811.11008.pdf" rel="noreferrer">https://arxiv.org/pdf/1811.11008.pdf</a></h1>

<p><a href="https://i.stack.imgur.com/PyBgx.png" rel="noreferrer"><img src="https://i.stack.imgur.com/PyBgx.png" alt="enter image description here"></a></p>

<h1>Thinking though, the <code>NP JJ</code> isn't the ultimate goal; the ultimate goal is to produce the <code>UP</code> or <code>DOWN</code> label based on some heuristics.</h1>

<p>Lets rephrase the steps:</p>

<ol>
<li><p>Parse the sentence with a <strong>parser</strong> (in this case <em>regular expression parser using some sort of grammar</em>)</p></li>
<li><p>Identify signal that the sentence has <strong>a pattern</strong> that can tell use about the ultimate label.</p>

<p>2a. Traverse the parse tree to extract <strong>another pattern</strong> that tells us about the performance indicator and numeric values. </p>

<p>2b. Use the extracted extracted numeric values to determine the directionality <code>UP</code> / <code>DOWN</code> using <strong>some heuristics</strong></p>

<p>2c. Tag the sentence with the <code>UP</code> / <code>Down</code> identified in (2b)</p></li>
</ol>

<h1>Lets see which component we can build first.</h1>

<blockquote>
  <p>2b. extract <strong>another pattern</strong> that tells us about the performance indicator and numeric values.</p>
</blockquote>

<p>We know the output to some percentage is always <code>CD NN</code> from </p>

<pre><code>('8.3', 'CD'), ('%', 'NN')
('11.8', 'CD'), ('%', 'NN')
</code></pre>

<p>So lets try catching that in the grammar. </p>

<pre><code>patterns = """
PERCENT: {&lt;CD&gt;&lt;NN&gt;}
"""

PChunker = RegexpParser(patterns)
PChunker.parse(pos_tag(word_tokenize(text)))
</code></pre>

<p>[out]:</p>

<pre><code>Tree('S', [('Operating', 'NN'), ('profit', 'NN'), ('margin', 'NN'), ('was', 'VBD'), 
  Tree('PERCENT', [('8.3', 'CD'), ('%', 'NN')]), 
(',', ','), ('compared', 'VBN'), ('to', 'TO'), 
  Tree('PERCENT', [('11.8', 'CD'), ('%', 'NN')]), 
('a', 'DT'), ('year', 'NN'), ('earlier', 'RBR'), ('.', '.')])
</code></pre>

<p>Now, how do we get this:</p>

<blockquote>
  <ol start="2">
  <li>Identify signal that the sentence has a pattern that can tell use about the ultimate label.</li>
  </ol>
</blockquote>

<p>We know that <code>&lt;PERCENT&gt; compared to &lt;PERCENT&gt;</code> is a good pattern, so lets try to encode it. </p>

<p>We know that <code>compared to</code> has the tags <code>VBN TO</code> from </p>

<pre><code> ('8.3', 'CD'),
 ('%', 'NN'),
 (',', ','),
 ('compared', 'VBN'),
 ('to', 'TO'),
 ('11.8', 'CD'),
 ('%', 'NN'),
</code></pre>

<p>How about this:</p>

<pre><code>patterns = """
PERCENT: {&lt;CD&gt;&lt;NN&gt;}
P2P: {&lt;PERCENT&gt;&lt;.*&gt;?&lt;VB.*&gt;&lt;TO&gt;&lt;PERCENT&gt;}
"""

PChunker = RegexpParser(patterns)
PChunker.parse(pos_tag(word_tokenize(text)))
</code></pre>

<p>[out]:</p>

<pre><code>Tree('S', [('Operating', 'NN'), ('profit', 'NN'), ('margin', 'NN'), ('was', 'VBD'), 
           Tree('P2P', [
               Tree('PERCENT', [('8.3', 'CD'), ('%', 'NN')]), 
               (',', ','), ('compared', 'VBN'), ('to', 'TO'), 
               Tree('PERCENT', [('11.8', 'CD'), ('%', 'NN')])]
               ), 
           ('a', 'DT'), ('year', 'NN'), ('earlier', 'RBR'), ('.', '.')]
    )
</code></pre>

<h1>But that pattern could have been any arbitrary number. We need a signal for the <code>performance indicator</code></h1>

<p>Since I'm no domain expert in the financial domain, simply using the existence of <code>operating profit margin</code> might be a good signal, i.e. </p>

<pre><code>from nltk import word_tokenize, pos_tag, RegexpParser

patterns = """
PERCENT: {&lt;CD&gt;&lt;NN&gt;}
P2P: {&lt;PERCENT&gt;&lt;.*&gt;?&lt;VB.*&gt;&lt;TO&gt;&lt;PERCENT&gt;}
"""

PChunker = RegexpParser(patterns)


text = "Operating profit margin was 8.3%, compared to 11.8% a year earlier."

indicators = ['operating profit margin']
for i in indicators:
    if i in text.lower():
        print(PChunker.parse(pos_tag(word_tokenize(text))))
</code></pre>

<p>[out]:</p>

<pre><code>(S
  Operating/NN
  profit/NN
  margin/NN
  was/VBD
  (P2P
    (PERCENT 8.3/CD %/NN)
    ,/,
    compared/VBN
    to/TO
    (PERCENT 11.8/CD %/NN))
  a/DT
  year/NN
  earlier/RBR
  ./.)
</code></pre>

<h1>Now how do we get the <code>UP</code> / <code>DOWN</code>?</h1>

<blockquote>
  <p>2b. Use the extracted extracted numeric values to determine the directionality UP / DOWN using some heuristics</p>
</blockquote>

<p>Just from the example sentence, other than "earlier" nothing else tells us about antecedence of the numbers. </p>

<p>So lets hypothesize this, if we have the pattern <code>PERCENT VBN TO PERCENT earlier</code>, we say that the 2nd percent is an older number.  </p>

<pre><code>import nltk
from nltk import word_tokenize, pos_tag, RegexpParser

patterns = """
PERCENT: {&lt;CD&gt;&lt;NN&gt;}
P2P: {&lt;PERCENT&gt;&lt;.*&gt;?&lt;VB.*&gt;&lt;TO&gt;&lt;PERCENT&gt;&lt;.*&gt;*&lt;RBR&gt;}
"""

def traverse_tree(tree, label=None):
    # print("tree:", tree)
    for subtree in tree:
        if type(subtree) == nltk.tree.Tree and subtree.label() == label:
            yield subtree

PChunker = RegexpParser(patterns)

parsed_text = PChunker.parse(pos_tag(word_tokenize(text)))
for p2p in traverse_tree(parsed_text, 'P2P'):
    print(p2p)
</code></pre>

<p>[out]:</p>

<pre><code>(P2P
  (PERCENT 8.3/CD %/NN)
  ,/,
  compared/VBN
  to/TO
  (PERCENT 11.8/CD %/NN)
  a/DT
  year/NN
  earlier/RBR)
</code></pre>

<h1>And the <code>UP</code> / <code>DOWN</code> label?</h1>

<pre><code>import nltk
from nltk import word_tokenize, pos_tag, RegexpParser

patterns = """
PERCENT: {&lt;CD&gt;&lt;NN&gt;}
P2P: {&lt;PERCENT&gt;&lt;.*&gt;?&lt;VB.*&gt;&lt;TO&gt;&lt;PERCENT&gt;&lt;.*&gt;*&lt;RBR&gt;}
"""

PChunker = RegexpParser(patterns)


def traverse_tree(tree, label=None):
    # print("tree:", tree)
    for subtree in tree:
        if type(subtree) == nltk.tree.Tree and subtree.label() == label:
            yield subtree

def labelme(text):
    parsed_text = PChunker.parse(pos_tag(word_tokenize(text)))
    for p2p in traverse_tree(parsed_text, 'P2P'):
        # Check if the subtree ends with "earlier".
        if p2p.leaves()[-1] ==  ('earlier', 'RBR'):
            # Check if which percentage is larger. 
            percentages = [float(num[0]) for num in  p2p.leaves() if num[1] == 'CD']
            # Sanity check that there's only 2 numbers from our pattern.
            assert len(percentages) == 2
            if percentages[0] &gt; percentages[1]:
                return 'DOWN'
            else:
                return 'UP'

text = "Operating profit margin was 8.3%, compared to 11.8% a year earlier."

labelme(text)
</code></pre>

<h1>Now the question begets...</h1>

<p>**Do you want to write so many rules and catch them using the <code>labelme()</code> above? **</p>

<p><strong>Are the patterns you write foolproof?</strong> </p>

<p>E.g. will there be a case that the pattern to compare percentages using the indicator and "earlier" will not be "UP" or "DOWN" as expected</p>

<p><strong>Why are we writing rules in the AI age?</strong> </p>

<p><strong>Do you already have humanly annotated data where there are sentences and their corresponding UP/DOWN labels?</strong> If so, let me suggest something like <a href="https://allennlp.org/tutorials" rel="noreferrer">https://allennlp.org/tutorials</a> or <a href="https://github.com/huggingface/transformers/blob/master/notebooks/03-pipelines.ipynb" rel="noreferrer">https://github.com/huggingface/transformers/blob/master/notebooks/03-pipelines.ipynb</a></p>
How to use NLTK Regex patterns to annotate financial news with UP/DOWN indicator?
<p>I'm working on replicating an algorithm describe in this paper: <a href="https://arxiv.org/pdf/1811.11008.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1811.11008.pdf</a></p>

<p>On the last page it describes extracting a leaf defined in the grammar labelled 'NP JJ' using the following example: Operating profit margin was 8.3%, compared to 11.8% a year earlier.</p>

<p>I'm expecting to see a leaf labelled 'NP JJ' but I'm not. I'm tearing my hair out as to why (relatively new to regular expressions.)  </p>

<pre><code>def split_sentence(sentence_as_string):
    ''' function to split sentence into list of words
    '''
    words = word_tokenize(sentence_as_string)

    return words

def pos_tagging(sentence_as_list):

    words = nltk.pos_tag(sentence_as_list)

    return words

def get_regex(sentence, grammar):

    sentence = pos_tagging(split_sentence(sentence));

    cp = nltk.RegexpParser(grammar) 

    result = cp.parse(sentence) 

    return result


example_sentence = "Operating profit margin was 8.3%, compared to 11.8% a year earlier."

grammar = """JJ : {&lt; JJ.∗ &gt; ∗}
            V B : {&lt; V B.∗ &gt;}
            NP : {(&lt; NNS|NN &gt;)∗}
            NP P : {&lt; NNP|NNP S &gt;}
            RB : {&lt; RB.∗ &gt;}
            CD : {&lt; CD &gt;}
            NP JJ : : {&lt; NP|NP P &gt; +(&lt; (&gt;&lt; .∗ &gt; ∗ &lt;) &gt;) ∗ (&lt; IN &gt;&lt; DT &gt; ∗ &lt; RB &gt; ∗ &lt; JJ &gt; ∗ &lt; NP|NP P &gt;) ∗ &lt; RB &gt; ∗(&lt; V B &gt;&lt; JJ &gt;&lt; NP &gt;)∗ &lt; V B &gt; (&lt; DT &gt;&lt; CD &gt;&lt; NP &gt;) ∗ &lt; NP|NP P &gt; ∗ &lt; CD &gt; ∗ &lt; .∗ &gt; ∗ &lt; CD &gt; ∗| &lt; NP|NP P &gt;&lt; IN &gt;&lt; NP|NP P &gt;&lt; CD &gt;&lt; .∗ &gt; ∗ &lt;, &gt;&lt; V B &gt; &lt; IN &gt;&lt; NP|NP P &gt;&lt; CD &gt;}"""

grammar = grammar.replace('∗','*')

tree = get_regex(example_sentence, grammar)

print(tree)
</code></pre>