May the Fourth: VADER for Credit Sentiment?

In a previous blog (here), we showcased the use of Snorkel for building a simple yet effective sentiment engine for credit.

As a brief reminder, Snorkel is a system for labelling data, at scale, which is much needed in order to use advanced machine learning models.

In the previous blog, we designed a couple of weak labelers in the form of

  • string lookup from a polarized dictionary,
  • regular expressions.

We didn’t use any readily available sentiment model.

In this blog, we will check whether VADER can be used as yet another weak labelers in the “data programming” Snorkel approach.

Before any experiments, it is not clear that VADER can help at all since:

VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media, and works well on texts from other domains.

and looking at the lexicon, one can find (the following lines have the format token, polarity and intensity, standard deviation in the scores given by human labelers, the 10 scores given by 10 human labelers):

$: -1.5 0.80623 [-1, -1, -1, -1, -3, -1, -3, -1, -2, -1]

%) -0.4 1.0198 [-1, 0, -1, 0, 0, -2, -1, 2, -1, 0]

%-) -1.5 1.43178 [-2, 0, -2, -2, -1, 2, -2, -3, -2, -3]

(8 2.6 1.0198 [4, 2, 1, 3, 3, 3, 3, 1, 2, 4]

1337 2.1 1.13578 [3, 1, 4, 0, 2, 3, 1, 2, 2, 3]

:’( -2.2 0.74833 [-2, -1, -2, -2, -2, -2, -4, -3, -2, -2]

:$ -0.2 1.83303 [-2, -1, 0, 0, -1, 1, 4, -3, 1, -1]

amongst the first tokens of the lexicon.

One can wonder if it is of any use to analyze market comments.

For this experiment, we will use credit focused blogs published daily by Hellebore Capital portfolio managers. These blogs can be found there on the DataGrapple platform.

import re
import pickle
import numpy as np
import pandas as pd
from scipy import sparse
from sklearn.metrics import confusion_matrix, f1_score

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

from metal.label_model import LabelModel
from metal.analysis import lf_summary, label_coverage
from metal.label_model.baselines import MajorityLabelVoter

with open('./blogs', 'rb') as file:
    blogs = pickle.load(file)
print("We consider for the in-sample", len(blogs), "blogs.")

blogs = pd.DataFrame([blog['title'].replace('\t', '') 
                   + ' ' + blog['content'].replace('\t', '').replace('\n', '').replace('\r', '')
                   for blog in blogs],
                  index = [i for i in range(len(blogs))])
We consider for the in-sample 1238 blogs.
0 That Is A Big Deal In a decently risk-on sessi...
1 Only Game In Town Today, the ECB pretty much d...
2 Impairment Bites HEMA (short for 4 unpronounce...
3 On The Red Today the 5y CDS of Crown Resorts L...
4 Shipping Names Rocked Today CMA CGM (CMACG) an...

Weak labelers

We consider the same weak labelers as the previous related blog as a baseline, and we will add VADER as yet another weak labeler.

For VADER, we will consider the ‘compound’ score computed on the whole blog. If its compound sentiment is strongly positive or negative, then we consider a POSITIVE or NEGATIVE label respectively, otherwise the VADER weak labeler abstains from labeling the blog.


def vader_sentiment(text):
    analyzer = SentimentIntensityAnalyzer()
    vs = analyzer.polarity_scores(text)
    if vs['compound'] > 0.8:
        return POSITIVE
    elif vs['compound'] < -0.8:
        return NEGATIVE
        return ABSTAIN

PERFORMING = r"""\b(\d+bps tighter|tighter by \d+bp|credit spreads tighten across the board|back to the lowest spread level|CDS tightens back|little appetite to bid for single-name protection|stock up|strong performance|spreads tightening|performed|tighter|tighten|beating expectations|best performers|best performing|outperformance|outperforming|outperformer|outperformers)"""
def contains_performing_expressions(text):
    return POSITIVE if, text) else ABSTAIN

GOOD_RATINGS = r"\b(S&P upgraded|upgrade|upgraded|upgraded by Fitch|upgraded by Moody's)"
def contains_upgrade_expressions(text):
    return POSITIVE if, text) else ABSTAIN

GOOD_MOODS = r"\b(reassured credit investors|good short-term option|risk-on|positively in credit|dovish|guidance was positive|good for credit|issues have been pushed back|bullish)"
def contains_good_mood_expressions(text):
    return POSITIVE if, text) else ABSTAIN

GOOD_LIQUIDITY = r"\b(resolve the liquidity issue)"
def contains_good_liquidity_expressions(text):
    return POSITIVE if, text) else ABSTAIN

BAD_MOODS = r"\b(risk-off|tough test|disappointed|continued deterioration|challenging for credit|brutal punishment|hawkish|profit warning|dampen credit outlook|bearish)"
def contains_bad_mood_expressions(text):
    return NEGATIVE if, text) else ABSTAIN

UNDERPERFORMING = r"""\b(cut its profit forecast|stocks fall|higher leverage|shares plunged|widened \d+bp|CDS widened c\d+bp|bonds fell roughly \d+pts|stock got crushed|quarterly profit miss|shares sunk|loses money|risk premium through the roof|stock lost|revenues declined|downtrend in revenue|Q[1-4] results missed|bonds were trashed|defaulted on its debt|survival of the company is under threat|lost its leadership|Q[1-4] sales missed|weaker demand|sales down|stocks declined|bid single-name protection|weakens credit metrics|profit warnings|guidance dropped|missed the estimates|worst-performing|widening|underperformers|widen +\d+bp|under more pressure|curve is inverted|worst performing|CDS is wider|underperforming|underperformed|bonds were down|CDS widen by c\d+bp)"""
def contains_underperforming_expressions(text):
    return NEGATIVE if, text) else ABSTAIN

BAD_RATINGS = r"\b(Fitch downgraded|outlook to negative|downgrade|downgraded|outlook at negative)"
def contains_downgrade_expressions(text):
    return NEGATIVE if, text) else ABSTAIN

FRAUDS = r"\b(money laundering|scandal)"
def contains_fraud_expressions(text):
    return NEGATIVE if, text) else ABSTAIN

DEFAULTS = r"\b(filed for bankruptcy|chapter 11|filed for creditor protection|continue as a going concern)"
def contains_default_expressions(text):
    return NEGATIVE if, text) else ABSTAIN

BAD_MOMENTUM = r"\b(risk premium has tripled)"
def contains_bad_momentum_expressions(text):
    return NEGATIVE if, text) else ABSTAIN

CATASTROPHE = r"\b(devastating impact|struck by hurricane)"
def contains_catastrophe_expressions(text):
    return NEGATIVE if, text) else ABSTAIN

LFs = [

LF_names = [

Evaluation of the weak labelers

Next step is to evaluate each weak labeler in terms of coverage, accuracy, contradictions and overlaps with the other weak labelers.

Ideally, we want to reach high global coverage as this set of weak labelers will be used to label the whole corpus we have for training.

def make_Ls_matrix(data, LFs):
    noisy_labels = np.empty((len(data), len(LFs)))
    for i, row in data.iterrows():
        for j, lf in enumerate(LFs):
            noisy_labels[i][j] = lf(row.values[0].lower())
    return noisy_labels

Below I load the “true” labels I did by reading 130 blogs in order to have

  • an evaluation set for the weak labelers,
  • a training set for the combination of these weak labelers into a strong labeler.

N.B. This strong labeler is not the final classifier model, it is just an intermediate step to have cleaner labels on all the texts of the corpus.

# Get true labels for LF set
with open('labels_for_training_labelling', 'rb') as file:
    labels = pickle.load(file)
print('Number of manually labeled examples:', len(labels),
      '\nRatio of blogs labeled:', round(len(labels) / len(blogs), 2),
      '\nRatio positive:', round(len([val for key, val in labels.items() if val == 1]) / len(labels), 2))
Number of manually labeled examples: 130 
Ratio of blogs labeled: 0.11 
Ratio positive: 0.42
# We build a matrix of LF votes for each DG blog
LF_matrix = make_Ls_matrix(blogs.iloc[:len(labels)], LFs)
Y_LF_set = np.array([labels[i] for i in range(len(labels))])

j Polarity Coverage Overlaps Conflicts Correct Incorrect Emp. Acc.
vader 0 [1.0, 2.0] 0.430769 0.376923 0.130769 50 6 0.892857
performing 1 1 0.492308 0.407692 0.223077 43 21 0.671875
upgrade 2 1 0.023077 0.015385 0.000000 3 0 1.000000
good_mood 3 1 0.146154 0.146154 0.092308 15 4 0.789474
good_liquidity 4 1 0.007692 0.007692 0.000000 1 0 1.000000
underperforming 5 2 0.353846 0.276923 0.130769 36 10 0.782609
downgrade 6 2 0.076923 0.061538 0.015385 10 0 1.000000
bad_mood 7 2 0.130769 0.130769 0.046154 14 3 0.823529
fraud 8 2 0.030769 0.030769 0.023077 4 0 1.000000
default 9 2 0.023077 0.000000 0.000000 3 0 1.000000
bad_momentum 10 2 0.007692 0.007692 0.007692 1 0 1.000000
catastrophe 11 2 0.007692 0.007692 0.000000 1 0 1.000000

We can observe that the VADER weak labeler performs surprisingly great on the development set: It has the second best coverage (43%), and it has an accuracy of 89% (which is the best one when excluding the very low coverage rules).

print("Overall, we reach a coverage ratio of", round(label_coverage(LF_matrix), 2), "over the training subset.")

LF_matrix = make_Ls_matrix(blogs, LFs)
print("Overall, we reach a coverage ratio of", round(label_coverage(LF_matrix), 2), "over the whole training set.")
Overall, we reach a coverage ratio of 0.87 over the training subset.
Overall, we reach a coverage ratio of 0.81 over the whole training set.

The overall coverage is greatly improved from the previous study not using VADER:

Overall, we reach a coverage ratio of 0.82 over the training subset.

Overall, we reach a coverage ratio of 0.66 over the whole training set.

Comparison of a naive combination (majority vote) vs. Snorkel combination

Baseline: majority vote

mv = MajorityLabelVoter()
Ls_train = make_Ls_matrix(blogs.iloc[:len(labels)], LFs)

mv = MajorityLabelVoter(seed=42)
print("Basic majority vote labels performance:")
scores = mv.score((Ls_train, Y_LF_set), metric=['accuracy','precision', 'recall', 'f1'])
Basic majority vote labels performance:
Accuracy: 0.823
Precision: 0.738
Recall: 0.889
F1: 0.807
        y=1    y=2   
 l=1    48     17    
 l=2     6     59    

Compared to previously:

F1: 0.758

Training and evaluation of Snorkel strong labeler
Ls_train = make_Ls_matrix(blogs, LFs)

label_model = LabelModel(k=2, seed=42)
Computing O...
Estimating \mu...
Finished Training

print("Snorkel generated labels performance:")
Snorkel generated labels performance:
Accuracy: 0.846
Precision: 0.854
Recall: 0.759
F1: 0.804
        y=1    y=2   
 l=1    41      7    
 l=2    13     69    

Compared to previously:

F1: 0.727

Ls_train = make_Ls_matrix(blogs, LFs)

Y_train_ps = label_model.predict_proba(Ls_train)
array([[0.79409309, 0.20590691],
       [0.34469373, 0.65530627],
       [0.45152156, 0.54847844],
       [0.25849666, 0.74150334],
       [0.59675846, 0.40324154],
       [0.76897848, 0.23102152]])
from metal.contrib.visualization.analysis import (

                           title="Label Distribution")

Y_dev_ps = label_model.predict_proba(Ls_train[:len(labels)])
plot_probabilities_histogram(Y_dev_ps[:, 0],
                             title="Probablistic Label Distribution for POSITIVE")
plot_probabilities_histogram(Y_dev_ps[:, 1],
                             title="Probablistic Label Distribution for NEGATIVE")

We get relatively more observations with extreme confidence, which means that the model is more confident if a blog is positive or negative, than what we obtained in the previous study without using VADER.

Let’s check a few examples at random:

print(blogs.iloc[500][0], "\n\n", Y_train_ps[500])
In Mario We Trust It was a strong day right from the start, extending the trend that emerged after the Italian referendum. Since the wides reached early on Monday, iTraxx Main (ITXEB) has tightened by 8bps (it closed at 74bps tonight), iTraxx Crossover (ITXEX) by 25bps (it closed at 315.5bps tonight which is the series tights) and iTraxx Financial Senior by 13bps (it closed at 99.5bps tonight). One would have expected the move to happen after the ECB meeting which is taking place tomorrow. But even if market participants see it as the last chance for some volatility in 2016, it looks as though right now they are pricing a very benign outcome. It is also fairly obvious from the option market where implied volatility came down aggressively over the last few days. For ITXEB, options are pricing a total move of less than 5bps until the December expiry (which takes place in 2-week time), going into an important ECB meeting – during which investors expect to get some clues regarding the direction of the purchase program in 2017 - while previous ones have been the drivers of the biggest moves in credit so far this year. 

 [0.76897848 0.23102152]
print(blogs.iloc[700][0], "\n\n", Y_train_ps[700])

 [0.76897848 0.23102152]
print(blogs.iloc[1100][0], "\n\n", Y_train_ps[1100])
Basis Dynamics Through The Roll Single names CDS rolled on the 20th June as they do each quarter, and the most liquid maturities have been pushed by a quarter. For instance, the “on-the-run” 5 year maturity is now September 2019 while it was June 2019 a week ago. Credit indices on their side only roll twice a year, and the “on-the-run” 5 year investment grade index is still iTraxx Main S21 with a June 2019 maturity. On indices, on the 20th June, dealers just took things where they left it the day before and resume trading as if nothing had happened. On single names, dealers should have factored in their prices the maturity extension. But there is always a tendency to leave their 5 year quotes unchanged, or at least a tendency not to reflect fully the increase in risk premium coming from the 3 months added to the maturity of new on the run contracts. This is particularly true in a bullish environment, like the one which prevails at the moment. This grapple shows that this naturally led to an increase in the basis between credit indices and their theoretical values. 

 [0.88713694 0.11286306]

The three comments above are correctly labeled as positive for credit.

print(blogs.iloc[600][0], "\n\n", Y_train_ps[600])
Is Brexit Good For Some? Japan woke up this morning to the tune of M&A; after a three-day-week-end. Japanese telecom group SOFTBK ( SoftBank Group Corp  ) is offering a 43% premium to Friday’s close to acquire  ARM Holdings Plc , a UK mobile industry leader, for £24.3Bln. The bumper deal would be one of the largest European technology transactions and Softbank’s largest acquisition to date. ARM derives the vast majority of its revenues from Asia and North America with clients such as Apple, Samsung Electronics or Qualcomm, and should be relatively immune from Brexit. So the main impact of the coming UK’s exit from the EU at the moment is the roughly 12% fall of the British Pound against the Japanese Yen. It certainly makes the deal more attractive, but given the steady rise of ARM’s stock since the beginning of May and the hefty premium paid, the operation still means that SOFTBNK’s net debt to EBITDA could rise from 3.9 times to 4.6 times. No wonder investors pushed SOFTBNK’s 5 year risk premium 35bps wider to 155bps. 

 [0.41538462 0.58461538]

This one was correctly labeled as negative for (SOFTBNK) credit.

Let’s have a look at the top 5 most confident (blog, label) for each category:

print("POSITIVE for credit:", "\n")
pos_sorted_idx = np.argsort(Y_train_ps[:, 0])
print(pos_sorted_idx[-5:], "\n")
for i in range(5):
    idx = pos_sorted_idx[-i-1]
    print(blogs.iloc[idx][0], "\n")
    print('-' * 115)
POSITIVE for credit: 

[ 65 627 114 858  50] 

Sitting On A Gold Mine In a fairly risk-on session, European CDS tightened, led by high-betas, among which metal & mining companies. GLENLN (-9bp), MTNA (-12bp) and AALLN (-7bp) outperformed both the iTraxx Main (-2bp) and its fair value (-2.6bp). That relative outperformance is not only due to high beta. Solid production results at AALLN, RIO and BHP have boosted sentiment towards the sector. A general bullish price action in metals space helped too. Both base (+0.5%/+3.0%) and precious metals (+1.5%/+3.0%) have been boiled by a weaker dollar (DXY -0.7%). This reflects a broad optimism ahead of next week’s US-China discussions (Jan 30-31) and FOMC meeting (Jan 29-30) where officials might discuss bringing forward the end of the balance sheet unwind (WSJ). An effective de-escalation in trade tensions, coupled with a more dovish Fed and Chinese stimulus are supportive of the sector. But, the medium-term outlook might not be that clear. Miners’ performance remains closely tied to global growth, especially EM growth, as those countries are the biggest consumers of metals (China). That said, if a global recession was to happen in 2020, companies over-weighting precious metals over industrial ones would probably outperform the sector, as gold prices would probably rise. Today ABX 5y CDS is tighter -2bp, close to the tightest level in a decade, in mid 70s. Appetite for M&A;, reputational risk and transparency issues are other risks chronically on the cards, in the sector. 

[0.94243489 0.05756511]
Bull… Bull… Bull… Usually the business day following the 19th of June is all about roll. Indeed, this is the date when the maturity of “on the run” CDS is extended by a quarter. Today, when dealers started quoting 5y, they meant September 2020 maturity and no longer June 2020 as they did on Friday. Given that, one would expect the 5 year CDS, which is the risk premium you pay to protect yourself until the maturity of your contract against the default of the referenced entity, to go up. But on the back of new found optimism regarding Greece, which mainly hinged on the fact that they were not told to go its separate way during the day, it was as if the roll never took place. Not only on the run 5 year CDS were not wider, but they were effectively much tighter than last Friday. The bullishness of the market is also obvious when you look at the time series of any credit index. Today was the biggest one day move tighter in almost 3 years, actually since Mr Draghi delivered his “Whatever it takes” speech. 

[0.94243489 0.05756511]
Mind The Gap Today was the roll day for all iTraxx/CDX indices, except the CDX HY. This was the main focus, pushing traded volumes high (IG $46B Main $30B Xover $19B). The iTraxx Main roll was stable throughout the session (c9bp in 5s c7bp in 10s). In contrast, the iTraxx Xover roll traded in the mid -7s region during the European session but once the US came in, flows became one-way, pulling it below -9bp. That move let us feel like preemptively existing long-risk positions on Xover s29 got rolled onto the new series, despite the difference of spread between the two. The chart above might let us feel like credit is wider today because of the roll impact, however today was a clearly risk-on session which saw CDS indices tighten and US equity indices reach new record highs.  

[0.94243489 0.05756511]
Another Change In Wind Direction Today was the day where CSPP kicked off. That initially gave a positive tone to the credit market which lasted until mid-afternoon when the euro began to strengthen. That sent credit indices into reverse and they finally closed unchanged across the board. One name which participated in the early rally never looked back though. WINDIM (Wind) saw its 5 year risk premium tighten dramatically from 498bps to 453bps for reasons that have nothing to with the ECB buying bonds. This was all due to a story that came out late last night that suggested parties could be interested in buying redundant wireless frequencies and towers to create a 4th operator in Italy. This would go a long way toward alleviating concerns of the regulator. By lining up buyers, Hutchison’s 3 Italia and Vimpelcom’s Wind are trying to avoid a repeat of last month, when the EU vetoed the merger between O2 and 3 in the UK. As investors are getting more bullish about a deal getting through, WIND’s 5 year CDS’s is changing path yet another time.  

[0.94243489 0.05756511]

[0.94243489 0.05756511]
print("NEGATIVE for credit:", "\n")
neg_sorted_idx = np.argsort(Y_train_ps[:, 1])
print(neg_sorted_idx[-5:], "\n")
for i in range(5):
    idx = neg_sorted_idx[-i-1]
    print('-' * 115)
NEGATIVE for credit: 

[ 52 757 190 769  47] 

A Tax Break Is Not Enough The market was fairly confident going into the FOMC meeting, and risk enjoyed a positive session across the board. There were a few outliers and things did not pan out very well for PEMEX’s (Petroleos Mexicanos) credit investors. The company has seen oil output decline every year since 2004 and is Latin America’s most indebted borrower. It loses money on its inefficient refining business and because of endemic fuel theft – the company reports as many as 41 illegal pipeline taps every day -, which led to the tragedy that killed more than 100 people 10 days ago. Yesterday night Fitch downgraded Mexico’s oil producer to one notch above junk despite the government proposed  $3.5Bln cut from its taxes. The downgrade to BBB- reflects the “continued deterioration of PEMEX’s standalone credit profile” and the “under-investment in the company’s upstream business” according to Fitch. They added that the company had been technically insolvent since 2009 by having a negative total equity balance. PEMEX’s 5-year risk premium jumped 29bps to 315bps, off the 270bps tights reached a couple of days ago. 

[0.01113657 0.98886343]
Another Day, Another Probe Fraud is a risk that cannot be modelled and is what can make credit a really asymmetrical asset in the most extreme instances. It is always damaging for a company’s image and leaves analysts questioning the reliability of the information they are given. But more often than not, when fraud is revealed at a company, it only represents a dent in the representation of the 5 year risk premium over time. It can be a really severe one like for VW ( Volkswagen AG ) this summer when the emissions scandal surfaced, or a milder one like for TSCO ( Tesco Plc ) last fall. At the moment, it looks as if the widening of IBM’s ( International Business Machine Corp ) 5-year-CDS will be contained following the disclosure yesterday that the SEC is looking into the accounting treatment of certain transactions in the US, the UK and Ireland. Apparently accounting for emerging technologies can be challenging, and so far investors have decided to (sort of) give IBM the benefit of the doubt, only pushing the 5 year risk premium 15bps wider at 59bps. 

[0.01572778 0.98427222]
Back On The radar For All The Wrong Reasons The last time the credit market really paid attention the sovereigns was a while ago, but this week was all about Emerging Markets and the sovereigns included in the CDXEM. Early on Argentina was under the spotlight after the country had to hike short term interest rates by almost 10 percentage points to stem a currency crisis as the country’s fiscal woes remain unresolved. The government had to bite the bullet and turned to the IMF to demand a $30Bln credit line. The IMF appears to be showing good will to pursue Argentina’s request and that stopped the rot to a certain extent, but investors are waiting for the terms and date of the deal. Argentina’s 5-year risk premium was pushed 72bps wider at 404bps. The Lebanese Republic, Turkey and Egypt all suffered from the renewed tension in the Middle East after America pulled out of the Iran nuclear deal and saw their 5-year risk premia rise 45bps to 570bps, 5bps to 237bps and 10bps to 320bps respectively. Malaysia was also under pressure – its 5-year CDS widened 14bps to 92bps - after the shock defeat in the general election of Prime Minister Najib Razak. It put into question the future of state-owned investment fund 1MDB which has been involved in a money laundering scandal. 

[0.01572778 0.98427222]

[0.01572778 0.98427222]
Nice Timing Last Friday, TITIM ( Telecom Italia SpA ) issued a profit warning. They announced that organic EBITDA of its domestic business unit is expected to be lower than last year. Overall, including TIM’s brasil performance, consolidated organic EBITDA is expected to be about €8.1Bln. Adjusted consolidated net financial debt is expected to be about €25.2Bln. If the debt number was globally in line with expectations, the preliminary EBITDA disappointed. TTIM’s press release also suggested that this measure would remain under substantial pressure at least throughout the first semester, and that the cost of protecting the investor base in Italy comes at a high cost. There are also question marks regarding the strategy of the struggling former monopoly which has still not decided on a plan for its landline network, its most valuable asset. Its proposal to legally separate the network from the rest of the company in order to set-up the business for potential deals was rejected by Italy’s communications regulator over the week-end. TITIM’s stock got crushed (it was suspended at some point during the day to stop the rout) and ended the session down roughly 6%. The price action in credit was not different, and TITIM’s 5-year risk premium widened 19bps to 348bps, its widest level in more than 6 years. The bonds the company issued a couple of weeks ago at 99.436 were trading at 98.88. Nice timing. 

[0.01909513 0.98090487]