# Convolutional Neural Network Model for Diabetic Retinopathy Feature
Extraction and Classification

Sharan Subramanian<EMAIL_ADDRESS>
Merrill F. West High School
Tracy, California 95376  Leilani H. Gilpin
<EMAIL_ADDRESS>
Computer Science and Engineering at UC Santa Cruz
Santa Cruz  California 95064

###### Abstract

The application of Artificial Intelligence in the medical market brings up
increasing concerns but aids in more timely diagnosis of silent progressing
diseases like Diabetic Retinopathy. In order to diagnose Diabetic Retinopathy
(DR), ophthalmologists use color fundus images, or pictures of the back of the
retina, to identify small distinct features through a difficult and time-
consuming process. Our work creates a novel CNN model and identifies the
severity of DR through fundus image input. We classified 4 known DR features,
including micro-aneurysms, cotton wools, exudates, and hemorrhages, through
convolutional layers and were able to provide an accurate diagnostic without
additional user input. The proposed model is more interpretable and robust to
overfitting. We present initial results with a sensitivity of 97% and an
accuracy of 71%. Our contribution is an interpretable model with similar
accuracy to more complex models. With that, our model advances the field of DR
detection and proves to be a key step towards AI-focused medical diagnosis.

##  1 Introduction

Medical diagnosis serves as the foundational step in patient care, determining
the pathway for treatment and intervention. According to Dr. Edmund Arthur of
the University of Alabama at Birmingham School of Optometry, the accuracy and
timeliness of diagnoses are essential, as they directly influence the
patient’s prognosis and quality of life. A delay in identifying a condition
can cause health issues, increase costs, and lower chances of full recovery.
Recognizing a disease in its nascent stages often provides a broader array of
treatment options and increases the likelihood of a favorable outcome.
Diabetic Retinopathy (DR) exemplifies conditions where early detection can
make a profound difference. DR, a complication resulting from diabetes,
threatens the retina’s blood vessels, jeopardizing the crucial light-sensitive
layer at the back of the eye. Despite the potential to prevent up to 98% of
vision loss with timely intervention, its silent progression often eludes
timely detection, resulting in irreversible vision impairments.

Artificial Intelligence (AI) has emerged as a solution for more effective
diagnostic tools. Within AI, there is a subset of complex models called
Convolutional Neural Networks (CNNs). However, CNNs applications for DR
therapy have been of high complexity, requiring a high amount of computational
resources. The most widely used approach for DR diagnosis consists of a
dilated eye exam administered by an ophthalmologist or optometrist. However,
deep learning models perform with the same accuracy as medical professionals
at processing and analyzing fundus images111Charters, L. (2023). Finding a
place for AI, machine learning in retinal imaging.
https://www.ophthalmologytimes.com/view/finding-a-place-for-ai-machine-
learning-in-retinal-imaging, which depict the retina in detail. CNNs may also
be able to diagnose other conditions, including cataracts, glaucoma, and even
illnesses outside the retina. In Figure 1, there are four main features from a
fundus image showing positive signs of DR. By processing these images, CNNs
can identify early indicators of DR, illuminating proactive medical
interventions aided by AI.

Figure 1: Different features of a CFP show the presence of DR. This Fundus
Image was pulled from the Messidor-2 dataset and had a classified severity of
6 (PDR). These are the 4 features that were extracted and processed by our
machine-learning model.

In this paper, we present the different models currently available for DR
detection and their limitations and challenges. We review the technical
methods used, including an in-depth explanation of the CNN architecture, our
new methodology and model, and an open-source GitHub repository for public
access222Github repository with the code for the proposed CNN model. This
repository also has a detailed readme file on how to set up and use the model.
https://github.com/s21sharan/CNN_DR_Detection. The training process and post-
training techniques used to improve accuracy are thoroughly examined in the
Explanation of CNN Layers and Methods sections, respectively. In addition to
the methodology, we compare pre-trained models with the proposed ADL-CNN model
and examine the benefits and disadvantages of each. Furthermore, we explore
the potential implications of CNN advancements in healthcare, suggest areas
for further research, and discuss the impact these innovations can have on
disease detection and management.

##  2 Literature Review

We provide a brief history of DR detection and how its diagnosis evolved. In
addition to going over the various pre-existing studies on CNN models for DR
detection, we also highlight each model’s and method’s limitations. Then, we
will highlight what our model brings that is different from other studies and
the advantages our model entails.

###  2.1 Historical overview of DR detection

In the early stages of DR diagnosis, its approach was based on a patient’s
symptomatic presentation and rudimentary examination techniques [1]. Eduard
Jaeger and Albert von Graefe, in the 1850s described diabetic macular changes:
a historical review suggests the evolution of our understanding[1]. It took
more than a century for a consensus to emerge regarding the link between
cystoid macular edema, which is caused by the accumulation of fluids in the
macula, and diabetes mellitus (DM), where the body’s immune system attacks and
destroys insulin-producing cells in the pancreas. Recognizing the connection
between cystoid macular edema and diabetes mellitus today influences
diagnostic methods for patients with visual symptoms.

The first line of diagnosis typically involved direct ophthalmoscopy, a method
discussed in the historical evolution of DR diagnosis[2]. Digital fundus
photography reshaped DR screening in the late 20th and early 21st century.
Comparisons between singly, two-field, and three-field 45-degree Color Fundus
Photographs (CFP) showed varying sensitivities and specificities. These
advancements have modernized DR diagnosis, with varying sensitivities and
specificities observed across different Color Fundus Photograph methods.

With more advances in integrated AI and DR screening, the FDA approved an
autonomous AI device that used 45-degree digital CFP for DR detection called
the EyeArt AI system. On a side note, confocal scanning ophthalmoscopy emerged
as a technique that provided ultra-widefield imaging.

With these advancements, other challenges lie ahead on this path of innovation
and integration. The intricacies of integrating various technologies
seamlessly, ensuring universal accessibility to these advanced diagnostic
tools, and refining AI algorithms for increased accuracy are among the ongoing
challenges. While previous models may have offered solutions to specific
challenges, a collaborative effort across medical, technological, and
regulatory domains is essential for realizing the full potential of these
novel diagnostic approaches.

###  2.2 Specific Studies on DR Detection using CNNs

In a study by Yasashvini R. et al., researchers attempted to classify DR using
both CNN and hybrid deep CNNs[3]. Their methodology combined the strengths of
CNN architectures with a hybrid deep CNN via a mixture of different
classifiers. They hypothesized that by combining the power of both these mixed
classifiers, the accuracy of DR would increase, better accounting for the
variety of features and abnormalities present in DR-affected CFPs. Their
results were successful in achieving accuracy rates close to 100%.

Following the onset of high precision in DR-prediction models, Pratt et al.
introduced the task of predicting severity in DR diagnosis[4]. Their multi-
dimensional data supports severity prediction in CFPs, from retinal features
such as micro-aneurysms, exudates, and hemorrhages. Pratt and his team
developed a CNN architecture that underwent data augmentation to achieve this
dual objective. In Figure 2, we can see the architecture that Pratt et al.
used. Their CNN model classified the fundus images between the seven different
severities with a sensitivity of 95% and an accuracy of 75% on the validation
images. In our work, we augment their architecture’s weights and layers.

Figure 2: CNN architecture for a DR detection model. This model finds the
severity of DR through 7 different classifications of DR (No DR, Mild DR,
Moderate DR, Severe DR, Very Severe DR, PDR, and Advanced PDR).

Albahli & Yar’s research emphasized the customizability of CNNs in the medical
domain[5]. Albahli & Yar designed three deep-learning models to detect the
severity of diabetic retinopathy from retina images and determine its
potential progression to macular edema. Their main challenge was the limited
dataset they used: the Indian Diabetic Retinopathy Image Dataset (IDRiD). To
counteract this, they designed features such as Brightness, Color, and
Contrast enhancement, Color Jitters, and Contrast Limited Adaptive Histogram
Equalization to generate a broader range of images. Their study used pre-
trained models like ResNet50, VGG16, and VGG19 and customized them to DR
detection. Their results, after validation, showcased the potential of custom
CNNs, with each model yielding reasonable outputs.

On the other hand, the study by Shu-I Pao et al. used entropy images to
increase DR detection performance[6]. In this context, the entropy image is an
abstract representation computed using the green component of the CFP. By
preprocessing these entropy images using unsharp masking, the researchers were
able to enhance the clarity and definition of the features. The outcome was a
bi-channel CNN, which effectively utilized the features from the gray level
and the green component’s entropy images. This approach, while complex,
provided a comprehensive manner for improving the detection performance of
severe DR cases.

In summary, these studies show the immense potential and versatility of CNNs
in DR detection. Whether it is the combined powers of CNNs and hybrid
networks, the emphasis on severity classification, the adaptability of custom
models, or the complex manipulation of entropy images, the domain of DR
detection is witnessing fast progress and growth.

###  2.3 Limitations of Current Models

One explicit limitation emerges from the study by Yasashvini et al. While
integrating standard CNNs with hybrid deep convolutional networks might
enhance precision, it inadvertently adds to the model’s complexity. Such
sophisticated models often require vast computational resources, both in terms
of training and inference. Furthermore, the more complex a model is, the
higher the likelihood of overfitting, especially when the available dataset is
not large enough. Our proposed model is less complex and has repeatedly been
tested on overfitting curves. Overfit models might perform exceedingly well on
training data but falter when faced with unseen data, making them unreliable
for real-world applications. Testing the model with multiple data sets to
ensure uniform reliability in the real world is important.

In the research led by Pratt et al., the results were primarily based on one
specific dataset: the Kaggle dataset. The variability and uniqueness of
retinal images suggest that a model trained on one dataset might not
generalize well to images from different sources, patients of varied
demographics, or images captured using different equipment. This bias-related
limitation ties into the data scarcity in DR and fundus imaging.

Albahli’s research, which introduced custom CNN models, highlighted another
significant limitation related to data scarcity. The requirement to employ
image generation techniques like BCC enhancing, CJ, and CLAHE to expand their
dataset indicates the prevalent challenge of limited datasets in the field.
Having a limited dataset not only hampers the depth of training but also
raises concerns about the model’s ability to generalize to a broader range of
images. In our model, we have trained and tested it with a more comprehensive
dataset and have seen repeatable and reliable results with additional data
sets. Moreover, artificially augmenting data can sometimes introduce new
artifacts or fail to replicate the subtle nuances of natural images,
potentially leading to models that might misinterpret or overlook certain
features. Another question arises by introducing many data augmentation
techniques: whether pre-processing techniques mask significant data features.

Shu-I Pao and his team’s research brings forth the intricacies of
preprocessing. While the emphasis on entropy images derived from the green
component of fundus photographs and preprocessing techniques like unsharp
masking is innovative, it also adds layers of processing that might not be
feasible in real-world, time-sensitive scenarios. Such intricate preprocessing
might also introduce biases or inadvertently filter out certain relevant
features, limiting the model’s diagnostic capabilities. Our model sticks to
basic preprocessing to resize the image and extract key features without
modifying or filtering the actual image itself. Lastly, a collective
limitation observed from all these studies is the gap between theoretical
performance and real-world implementation. While these models showcase
impressive metrics in controlled environments, challenges such as
computational constraints, integration with existing healthcare systems,
interpretability of results, and ensuring consistent performance across
diverse patient populations remain. Addressing these practical limitations is
paramount for the successful and widespread adoption of CNNs in DR
diagnostics.

###  2.4 AI Trust in Healthcare Environment

The area of healthcare, particularly in recent years, has observed a large
fusion of artificial intelligence (AI) applications, covering a broad spectrum
ranging from diagnostics and clinical decision-making to digital public
health. AI’s potential to enhance clinical outcomes and efficiency is
acknowledged, but the acceptance of this technology remains contingent on
multiple factors.

During the COVID-19 pandemic, Chalutz Ben-Gal’s study investigated the
acceptance of AI in primary care[7]. It was found that while many patients
showcase a resistance towards AI, several factors can potentially tilt the
balance in favor of AI. Explicit mentions of AI’s superior accuracy, positive
nudges from primary care physicians, and assurances of personalized patient
listening experiences could enhance AI acceptance. Notably, Robertson et al.
found that specific demographics, including Black respondents and older
individuals, were less inclined to choose AI, highlighting the consideration
of demographic-specific concerns in AI adoption[8].

Diving deeper into the diagnostic field, the potential of AI to augment
diagnostic precision is immense. However, as noted by Robertson et al.,
patients’ trust in AI may waver based on disease severity and their perceived
personalization of the AI system. Lastly, considering the viewpoint of
healthcare professionals, Lambert et al. presented an integrative review
highlighting the barriers to AI acceptance in the hospital setting[9]. The
fear of losing professional autonomy in integrating AI into existing clinical
workflows was a recurring concern. Conversely, training geared towards AI
utilization emerged as a positive catalyst and gained acceptance among
countless professionals. The study further emphasized the significance of
involving end-users during the early stages of AI development, suggesting a
collaborative approach to ensure that the technology meets real-world needs
and challenges.

In conclusion, while the potential benefits of AI in healthcare are vast, its
acceptance remains an issue, necessitating a comprehensive understanding of
both patients’ and practitioners’ perspectives coupled with tailored
approaches to address their specific concerns.

##  3 Understanding Convolutional Neural Networks

Convolutional Neural Networks (CNNs) are pivotal in machine learning and
artificial intelligence, especially for image processing and classification
tasks. Inspired by biological neural networks, CNNs emulate human perception
by recognizing patterns from image pixels, enabling classification,
identification, and visual data reconstruction [10]. In the realm of
healthcare, particularly radiology, they exhibit considerable potential in
enhancing conventional diagnostic methodologies, resulting in improved speed,
reliability, and occasionally superior accuracy[11].

Fundamentally, neural networks comprise interconnected nodes or ”neurons” that
execute elementary mathematical operations. These networks are constructed to
model intricate data relationships. As depicted in Figure 3, CNNs encompass
input layers for data reception, hidden layers for data processing, and output
layers for prediction/decision-making. Unlike conventional neural networks
employing full interconnections between layers, CNNs employ sparse connections
to focus on local features, rendering them highly efficient in capturing
spatial hierarchies. Consequently, CNNs are exceptionally well-suited for
tasks such as image recognition, where the extraction of local features such
as edges, corners, and textures offers valuable information.

Figure 3: This flowchart highlights the CNN architecture that is used in our
proposed model. It has different kernels and depth layers that take input
images and output the classification. The input layer is where the color
fundus image is fed into the neural network. Then the convolutional layers
apply a set of filters to the input image, detecting specific features like
edges, textures, and shapes that are tied into the DR signs, as highlighted in
Figure 1. Then, the ReLU activation function is applied element-wise,
introducing nonlinearity. The weights represent the parameters that the
network learns during training, and the final output is the severity
classification of DR based on features extracted from the input image by
preceding layers.

CNNs belong to a category of neural networks known as Deep Learning
models[12]. As the name suggests, these models are characterized by their
depth or the number of layers data must pass hierarchically. Deep Learning, a
subfield of machine learning, offers significant advantages over traditional
machine learning methods. The depth of the network allows for more complex
features to be learned, thereby increasing the model’s ability to understand
intricate patterns in the data, especially in medical imaging tasks[13].

###  3.1 Breaking Down the Architecture: Layers, Filters, Feature Extraction,
etc.

The convolutional layer is the core aspect of a CNN. This layer performs the
convolution operation on the input, passing the result to the next layer.
These operations focus on local regions of the input, preserving the
relationship between pixels by learning image features[14]. For instance, in
DR detection, this localized focus enables CNNs to discern pixel-level
details, capturing critical retinal features and enhancing the model’s
feature-learning capability.

After convolution, the layers often employ an activation function, the most
common being the Rectified Linear Unit (ReLU). ReLU, in the kernels section of
Figure 3, introduces non-linearity into the system, which allows the model to
learn from the error and make adjustments, improving its performance during
the training process.

Following the convolutional and activation layers, CNNs usually contain
pooling layers to reduce dimensionality. The pooling operation condenses the
feature map obtained from the previous layer, usually by taking the maximum
(as in Equation 1) or average value (as in Equation 2) from a group of values
in the feature map333Nanos, G., & Nanos, G. (2023). Neural Networks: Pooling
layers, Baeldung on Computer Science. Baeldung on Computer Science.
https://www.baeldung.com/cs/neural-networks-pooling-layers. This pooling
process also results in reduced size and computational complexity of the
resultant feature map.

${MaxPooling}(X)_{i,j,k}=\max_{m,n}X_{i\cdot s_{x}+m,j\cdot s_{y}+n,k}$ (1)
${AvgPooling}(X)_{i,j,k}=\frac{1}{f_{x}\cdot f_{y}}\sum_{m,n}X_{i\cdot
s_{x}+m,j\cdot s_{y}+n,k}$ (2)

One of the significant operations in CNNs involves filter or kernel feature
extraction, as illustrated in Figure 3. Filters, which are small and
adjustable parameters, traverse the input image, generating feature maps. For
example, one filter may focus on edge detection, while another specializes in
identifying specific colors. The concept of ”stride” defines how filters move
across the input image, determining their step size. These adjustable
parameters collectively provide spatial control over the convolution
operation, enabling precise adjustments and customization.

Towards the end of the network, fully connected layers, also known as dense
layers, utilize the high-level features acquired from preceding layers to
classify the image into distinct categories. These layers share a structural
resemblance to a standard multi-layer perceptron neural network.

###  3.2 Explanation of CNN Layers, with a Specific Focus on Image
Classification Tasks

Forward propagation in CNNs entails the sequential transmission of the input
image through multiple layers. As the image traverses from one layer to the
next, it gives rise to feature maps characterized by reduced dimensionality,
which are subsequently employed in the final layers for classification
purposes[15]. Similar to other neural networks, CNNs use backpropagation for
training. This algorithm evaluates the disparity between the output produced
by forward propagation and the expected outcome, subsequently modifying the
filter weights, as illustrated in Figure 3. This weight adjustment process is
often facilitated by Stochastic Gradient Descent (SGD) optimizers.

  1. 1.

Initialization: Initialize the weights ($w_{1}$, $w_{2}$) and biases
($b_{1}$,$b_{2}$) with random values.

  2. 2.

Forward Pass: Pass a training example ($x_{t}rain$) through the network to
compute the predicted output ($y$)

  3. 3.

Compute Loss: Calculate the loss, as in equation 3, using predicted output and
the true target

$L=\frac{1}{2}(y-y_{true})^{2}$ (3)

  4. 4.

Backpropagation: Compute the gradients of the loss with respect to the weights
and biases using the chain rule (calculus)

  5. 5.

Update Weight: Adjust the weights using the gradients and small learning rate
according to SGD update rule444Ruder, S. (2020). An overview of gradient
descent optimization algorithms. ruder.io. https://www.ruder.io/optimizing-
gradient-descent/, as in Equation 4 and 5

$w_{i}=w_{i}-\alpha(\frac{\vartheta L}{\vartheta w_{i}})$ (4)
$b_{i}=b_{i}-\alpha(\frac{\vartheta L}{\vartheta b_{i}})$ (5)

  6. 6.

Repeat: Repeat steps 2-5 for specified number of epochs until convergence

Pre-trained models are applied in various scenarios, particularly when working
with limited training datasets. This approach applies pre-trained CNNs, with
subsequent fine-tuning of their final layers for adaptation to new tasks. This
reduces computational resource demands instead of using significant compute to
train a CNN from scratch. CNNs utilize batch processing techniques, which
enhance training efficiency by enabling faster and more parallel computation.
Furthermore, cross-entropy loss functions are used to quantify the disparity
between predicted outputs and actual labels, with the primary training
objective being the minimization of this loss function. Without the cross-
entropy loss function, CNNs risk overfitting, introducing biases, and failing
to generalize effectively on unseen data. Similarly, a CNN’s performance can
be influenced by hyperparameters like learning rate, filter size, filter
count, and layer count. Consequently, hyperparameter optimization is
frequently conducted to refine these settings.

Beyond image classification, CNNs find applications in complex tasks such as
video analysis, natural language processing, and autonomous driving. In the
healthcare domain, their image recognition capabilities are invaluable for the
detection of anomalies in X-rays, MRIs, CT scans, and CFPs, as elaborated in
this paper. As CNNs and their associated technologies continue to evolve, they
hold immense promise for healthcare applications, ranging from early disease
classification for conditions like cataracts and cancers to real-time patient
monitoring.

##  4 Methodology

###  4.1 Data Sets Used

Dataset | Year Released | Image Count | Field of Vision (FOV) |
---|---|---|---|---
Kaggle | 2015 | 88k | 50° |
APTOS | 2019 | 13k | Not specified |
DDR | 2019 | 13.6k | 45° |
Messidor-2 | 2010 | 1748 | 45° |
DeepDRid | 2019 | 2256 | Not specified |

Table 1: The different publicly available CFP datasets for classifying DR.

We incorporated multiple data sets to develop, train, and validate our model,
with a primary focus on the Messidor-2 dataset. Table 1 gives a snapshot of
the datasets we utilized. The primary dataset which our model was built from
is the Messidor-2 dataset. Between October 2009 and September 2010, diabetic
patients were imaged using a Topcon TRC NW6 non-mydriatic fundus camera with a
45° field of view, resulting in 345 total DR examinations. Only macula-
centered images were considered for this dataset. This was then combined with
the Messidor-original database, which has a total of 529 DR examinations, to
create the Messidor-2 Data set.

Initially, we used a 70-30 train-test split with the Messidor-2 dataset taking
inspiration from other studies in the field[16]. However, we adjusted this
split to 80-20 to minimize training and validation losses.

Our secondary data sets include the Kaggle dataset, APTOS, DDR, and DeepDRid.

  1. 1.

Kaggle: This dataset was involved in one of the Kaggle Challenges relating to
medical imaging and CNNs. This dataset covers a diverse range of sources and
conditions, with over 88,000 images.

  2. 2.

APTOS (Asia Pacific Tele-Opthalmology Society): This society concentrates on
eye-related conditions, including DR. This dataset includes patients of Asian
descent and is very comprehensive, with over 13k images.

  3. 3.

DDR: This dataset is sourced from 147 hospitals spread across 243 provinces in
China. These images are classified based on the severity of diabetic
retinopathy into 5 categories.

  4. 4.

DeepDRid: This dataset originated from the 2nd Diabetic Retinopathy:
Segmentation and Grading Challenge in collaboration with ISBI in 2018. This
dataset focuses on three pivotal tasks: dual-view disease grading, image
quality estimation, and transfer learning.

All these datasets were classified based on the severity of DR present in the
images. The five severities were none, mild, moderate, severe, and
proliferative DR. The images of poor data quality were excluded from the
datasets, and a manual review took place before putting these images into our
model. In Figure 4, we can see examples of CFPs that passed this review and
made it onto the dataset. There are significant differences between these CFPs
and their features, including the hard exudate, cotton wool, and hemorrhage,
as annotated in Figure 1.

Figure 4: This figure has three images of CFPs that contain DR (in the top
row) and three images of CFPs that do not contain DR (in the bottom row). The
annotated features are as follows: a. Hemorrhage; b. MA; c. Drusen; d.
Exudates; e. Optic disc; f. Fovea; g. Blood vessel; h. Background doi:
https://doi.org/10.1371/journal.pone.0066730.g001

###  4.2 Methods

Our approach is split into two primary stages: the training/validation phase
and the testing phase.

First, Data was aggregated from various sources like hospitals and specialized
data repositories during the initial phase. We used the Messidor-2 dataset as
our baseline data. We removed any 25 low-quality images from the consolidated
dataset through an ancillary intelligent model and organized them using a
70-30 split555Our updated dataset is attached to the github link in Footnote
2.. We split each subcategory, or severity, using Sklearn’s train-test split
function.

During the pre-processing stage, we used two operations to reduce noise and
remove gaps in the background and foreground of the CFP. The first procedure
is known as Erosion. Erosion is used to eliminate or spike the edge of the
area and is represented by Equation 6[17].

$A\ominus B={p|B_{p}\subseteqq A}$ (6)

Then we also applied dilation, which is used to broaden the rim of the
background or foreground image configuration. This procedure helps us fill the
gap created through erosion and is defined by equation 7[17]:

$A\oplus B={x|B_{x}\cap X\neq 0}$ (7)

Where

$\ominus$

denote the dilation;

$\oplus$

denote the erosion; $A$ = Structuring element and $B$ = the erosion of the
dilation of that set. These equations are from the Tyler Coye Algorithm.

In the CNN model, we used convolutional operations to extract features. We
could convert the images into feature activation maps using diverse filter
sizes. Then, using the Rectified Linear Unit (ReLU) activation function, we
further enhanced this model with non-linearity. Following this, we employed a
pooling layer, an instrumental part in compressing the activation map’s
dimensions without significant information loss. We then vectored the
activation maps, processed through several fully connected layers, resulting
in a severity rating based on the DR-positive or DR-negative classification.

In the model’s second stage, we tested the data using our test-split of the
Messidor-2 data set and other secondary data sets, as highlighted in section
LABEL:Data_Sets_Used. To prevent our model from overfitting, we implemented
the early stopping mechanism, which was used to reduce training time from 8
hours 36 mins to 4 hours and 55 mins. This technique monitors our model’s
performance on the validation set and halts training once the performance
starts deteriorating. This ensures that the model does not overfit. We also
implemented a drop-out mechanism, which allowed randomly selected neurons to
be ignored at each iteration. This ensured that our neural network remained
accurate and generalized well to new-unseen data.

We also used the Keras auto-hyperparameter tuning module to tune crucial
parameters of our models. This algorithm uses a mathematical approach to
parameter tuning; this tweaks the learning rate, batch size, epochs, and
dropout rate to ensure the lowest validation losses and highest accuracy. We
were able to increase our validation accuracy from 55% to 70% using this
hyperparameter tuner.

##  5 Results

In our initial study, the DR model was trained, validated, and tested on the
Messidor data set. With 1748 images split into five severity categories, we
had about 300 for each severity. These 500 images were then split in an 80-20
ratio for the test-train split.

Figure 5: Figure 5a shows the Accuracy curves for our model as tested on the
Messidor-2 Data set. Figure 5b shows the Loss curves for our model as tested
on the Messidor-2 Data set.

As seen in Figure 5a, our model’s performance throughout the training process
is consistent. Averaging at approximately 97% over 100 epochs, this model can
recognize and predict patterns within the training dataset. The validation
accuracy remained at 70% throughout the 100 epochs. Though slightly lower than
the training data, this validation accuracy aligns with the understanding that
generalizing our model to unseen data will reasonably lower the accuracy.

The training loss, as seen in Figure 5b, remained under 0.3 across all epochs.
On the other hand, the validation loss was higher at about 1.5-1.8. As shown
in Table 2, the training and validation accuracy also stabilized around the
seventh epoch, hinting that our accuracies were not just a fluke.

Additionally, this model was reset and repeatedly tested, bringing stable and
similar accuracy values every time.

##  6 Discussion

Our model demonstrates the effectiveness of CNNs in DR detection, with a
streamlined architecture. In comparison to Yasashvini R. et al.’s hybrid deep
CNNs, our model strikes a balance between complexity and efficacy. Unlike
Albahli et al.’s research, which relied on multiple image enhancement
techniques due to data scarcity, our model was trained on a more comprehensive
dataset and validated on additional datasets.

In contrast to techniques like entropy-based image processing highlighted by
Shu-I Pao et al., our model aims for direct, effective, and reproducible DR
classification, maintaining competitive validation accuracy.

Epochs | Training Loss | Training Accuracy | Validation Loss | Validation Accuracy
---|---|---|---|---
0 | 0.075937 | 0.971781 | 1.577781 | 0.700000
1 | 0.076237 | 0.964727 | 1.689509 | 0.694737
2 | 0.083331 | 0.970018 | 1.463975 | 0.710526
3 | 0.076263 | 0.973545 | 1.697605 | 0.700000
4 | 0.96008 | 0.962963 | 1.778189 | 0.663158
5 | 0.073751 | 0.966490 | 1.582166 | 0.693684
6 | 0.072941 | 0.964727 | 1.623718 | 0.784211
7 | 0.076225 | 0.966490 | 1.609345 | 0.700000

Table 2: Training and Validation Results for Proposed CNN model

A 20% delta between training and validation metrics indicates room for
improvement. Future work includes implementing regularization techniques like
L1 or L2 regularization to address this disparity. Additionally, inspired by
Su-I Pao et al.’s use of entropy images, we may explore further feature
engineering techniques to extract informative features from retinal images.

In the evolving field of DR detection, our model offers promise and
scalability. Future iterations, along with advancements such as regularization
techniques, have the potential to improve accuracy. Our approach’s simplicity
enables integration with diverse datasets and applicability across
demographics and equipment sources.

##  7 Conclusion

In our study on DR detection using CNNs, we achieved enhanced efficiency and
accuracy in categorizing DR severity levels from retinal images. Our approach,
which incorporated preprocessing techniques like erosion and dilation
alongside convolutional operations, consistently yielded a 97% training
accuracy over 100 epochs. Impressively, our validation accuracy reached 71%,
marking significant progress towards automated medical diagnosis in DR.

Comparative analysis with other studies underscores the effectiveness of our
approach, striking a balance between model understanding and detection
efficacy. The model is adaptable across diverse datasets, requiring minimal
image enhancement due to the comprehensive Messidor-2 dataset used. However,
the gap between training and validation metrics indicates potential for model
enhancement, including advanced regularization techniques and feature
engineering to improve accuracy and mitigate overfitting.

Future improvements may explore more advanced preprocessing techniques and
incorporate regularization methods to bridge the gap between training and
validation loss metrics. Collaborating with healthcare institutions to obtain
diverse datasets could further enhance the model’s adaptability across various
clinical scenarios.

Our consistent results and the interpretability of our approach underscore the
reliability of CNNs in DR detection. Our work raises questions about how
healthcare professionals will adapt to AI technologies in the medical field
and the potential for more optimal Neural Networks in medical classification
tasks. As the medical community embraces the intersection of healthcare and
AI, the advancements presented in this and other studies contribute to a safer
and more clinically advanced society.

## References

  * [1] Wolfensberger, T. J., and Hamilton, A. M., 2001.  “Diabetic retinopathy–an historical review”.  Semin. Ophthalmol., 16(1), Mar., pp. 2–7.
  * [2] Matuszewski, D. J., Hast, A., Wählby, C., and Sintorn, I.-M., 2017.  “A short feature vector for image matching: The Log-Polar magnitude feature descriptor”.  PLoS One, 12(11), Nov., p. e0188496.
  * [3] Yasashvini, Raja Sarobin M., V., Panjanathan, R., Jasmine, G., and Anbarasi, J., 2022.  “Diabetic retinopathy classification using CNN and hybrid deep convolutional neural networks”.  Symmetry (Basel), 14(9), Sept., p. 1932.
  * [4] Pratt, H., Coenen, F., Broadbent, D. M., Harding, S. P., and Zheng, Y., 2016.  “Convolutional neural networks for diabetic retinopathy”.  Procedia Comput. Sci., 90, pp. 200–205.
  * [5] Albahli, S., and Nabi Ahmad Hassan Yar, G., 2022.  “Detection of diabetic retinopathy using custom CNN to segment the lesions”.  Intell. Autom. Soft Comput., 33(2), pp. 837–853.
  * [6] Pao, S.-I., Lin, H.-Z., Chien, K.-H., Tai, M.-C., Chen, J.-T., and Lin, G.-M., 2020\.  “Detection of diabetic retinopathy using bichannel convolutional neural network”.  J. Ophthalmol., 2020, June, p. 9139713.
  * [7] Chalutz Ben-Gal, H., 2022.  “Artificial intelligence (AI) acceptance in primary care during the coronavirus pandemic: What is the role of patients’ gender, age and health awareness? a two-phase pilot study”.  Front. Public Health, 10, p. 931225.
  * [8] Robertson, C., Woods, A., Bergstrand, K., Findley, J., Balser, C., and Slepian, M. J., 2023.  “Diverse patients’ attitudes towards artificial intelligence (AI) in diagnosis”.  PLOS Digit. Health, 2(5), May, p. e0000237.
  * [9] Lambert, S. I., Madi, M., Sopka, S., Lenes, A., Stange, H., Buszello, C.-P., and Stephan, A., 2023.  “An integrative review on the acceptance of artificial intelligence among healthcare professionals in hospitals”.  NPJ Digit. Med., 6(1), June, p. 111.
  * [10] Shin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., and Summers, R. M., 2016.  “Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning”.  IEEE Trans. Med. Imaging, 35(5), May, pp. 1285–1298.
  * [11] Yamashita, R., Nishio, M., Do, R. K. G., and Togashi, K., 2018.  “Convolutional neural networks: an overview and application in radiology”.  Insights Imaging, 9(4), Aug., pp. 611–629.
  * [12] Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., and Farhan, L., 2021.  “Review of deep learning: concepts, CNN architectures, challenges, applications, future directions”.  J. Big Data, 8(1), Mar., p. 53.
  * [13] Yala, A., Lehman, C., Schuster, T., Portnoi, T., and Barzilay, R., 2019.  “A deep learning mammography-based model for improved breast cancer risk prediction”.  Radiology, 292(1), July, pp. 60–66.
  * [14] Taye, M., Morrow, D., Cull, J., Smith, D. H., and Hagan, M., 2023.  “Deep learning for FAST quality assessment”.  J. Ultrasound Med., 42(1), Jan., pp. 71–79.
  * [15] Peng, J., Pedersoli, M., and Desrosiers, C., 2021.  “Boosting semi-supervised image segmentation with global and local mutual information regularization”.  J. Mach. Learn. Biomed. Imaging, 1(MIDL 2020), June, pp. 1–29.
  * [16] Mutawa, A. M., Alnajdi, S., and Sruthi, S., 2023.  “Transfer learning for diabetic retinopathy detection: A study of dataset combination and model performance”.  Appl. Sci. (Basel), 13(9), May, p. 5685.
  * [17] Sarki, R., Ahmed, K., Wang, H., Zhang, Y., Ma, J., and Wang, K., 2021.  “Image preprocessing in classification and identification of diabetic eye diseases”.  Data Sci. Eng., 6(4), Aug., pp. 455–471.
  * [18] https://doi.org10.1016/j.procs.2018.05.069.  Accessed: 2023-10-8.


# Information Content of Financial Youtube Channel: Case Study of 3PROTV and
Korean Stock Market

HyeonJun Kim∗
School of Finance, Soongsil University
<EMAIL_ADDRESS>or<EMAIL_ADDRESS>
###### Abstract

We investigate the information content of 3PROTV, a south Korean financial
youtube channel. In our sample we found evidence for the hypothesis that the
channel have information content on stock selection, but only on negative
sentiment. Positively mentioned stock had pre-announcement spike followed by
steep fall in stock price around announcement period. Negatively mentioned
stock started underperforming around the announcement period, with
underreaction dynamics in post-announcement period. In the area of market
timing, we found that change of sentimental tone of 3PROTV than its historical
average predicts the lead value of Korean market portfolio return. Its
predictive power cannot be explained by future change in news sentiment,
future short term interest rate, and future liquidity risk.

keywords: [Information Content], [YouTube], [Market Timing], [Stock Selection]

##  1 Introduction

An information material have information content, in Beaver(1968)’s term, ”if
it leads to a change in investors’ assessments of the probability distribution
of future returns (or prices), such that there is a change in equilibrium
value of the current market price”. [4] This argument implies that information
content causes future price moves and could be used for future return
prediction. There are trivial source of information contents like earning
reports or president’s letters to shareholders information. [1, 4]

In that sense, it will be somewhat abnormal for a normal YouTube content
creator to have an information of such impact. There are some studies that
investigates the relationship between social media reaction and stock price,
and few interpreted that it have information content. [12] However it is
questionable that this effects are because of whether mispricing caused by
sentiment, or change in permanent equilibrium value. Though it could be agreed
that its economical significance cannot be ignored. [11].

For more robust theoretical foundation, we can consider more satisfying
source, financial experts. For example, there is evidence that analyst reports
have information content even controlling for confounding firm events. [2] Our
subject in interest is at the mix of two extremes, social media content and
professional content. The platform of the content is social media, therefore
the influence of noise traders is inevitable. However, the content creators
are financial experts, which opens up the possibility of creating new
information by researching for undiscovered data or interpreting existing
data.

Here we focus on 3PROTV, one of the example of expert-run YouTube channel.
3PROTV, including its YouTube channel is expected to worth almost 200 million
dollars to date, and it has been one of the most rapidly grown Financial
YouTube channels in South Korea. It’s popularity had significantly grown
during COVID-19 pandemic period, when quantitative easing policies cause the
stock market to spike, which attracted new investors and demands for
investment information had risen. The impact of the channel’s content do seems
socially significant, however is unclear that it will have information about
the future stock market. In this paper, We investigate the latter.

The result of the analysis is quite unexpected. Roughly speaking, 3PROTV’s
content does have information content in our sample period, both on stock
selection and market timing. Stocks that are negatively mentioned by 3PROTV
had underperform after the announcement, while positively mentioned stock had
outperformed before the announcement. If we consider 3PROTV’s information as
sentimental, the result will be inconsistent with the prediction that current
literature like Baker and Wurgler (2007) suggest, which predicts negative
sentiment causes future outperformance, and vice versa. [3] Another surprising
part is that 3PROTV’s content sentiment, when preprocessed properly, can serve
as future return prediction indicator. We developed a sentiment index using
3PROTV’s content sentiment and found robust relation between future return and
the index regradless of index parameter. The daily trading strategy using our
sentiment index show outperformance from 11.5% to 23.6% without transaction
cost. The predictability persisted even controlling for future news sentiment,
short term interest rate, and liquidity risk factor ($PctZero$), making
3PROTV’s content information somewhat independent from those information. In
section 2 we show our methodology, and in section 3 we present our empirical
results. In section 4 we conclude the study with some remarks.

##  2 Methodology

For analyzing the 3PROTV’s specific content, we scraped YouTube’s auto-
transcription data to extract the mentioned stock and script’s overall
sentiment through August 2, 2022 to November 28, 2022, which in total a 500
content scripts. First, we concatenate the script within the same day and
tokenize, i.e. decompose the each content’s script such that the script
becomes a list of phased words. We then match the tokenized words of the
script with DataGuide’s Korean firm name list. We exclude some common noun-
like stock names like ”태양”(sun in Korean), ”대상”(subject, grand prize, etc. in
Korean), which might be mentioned because of other reasons. Another attempts
to lower the selection bias is to only select the stock name into our sample
only when the stock name is mentioned more than 2 times. By this method, we
ensure that those stocks represents the main theme of that day’s script, while
lowering the chance of a stock with low relevance to the content being
accidentally included in the sample.

For sentiment analysis, we use KOSELF(Korean Sentiment Lexicon for Finance)
[5] to extract the sentiment level of each 3PROTV’s content script. KOSELF is
a Korean Sentiment Lexicon exclusively developed for financial usage, meaning
that it will detect financial setiment in scripts better than other
alternatives like KOSAC or KNU’s sentiment lexicon, which are developed for
more general sentiment analysis. We use the sentiment level of content release
date $i$ $SENTIMENT^{(i)}$ , which is calculated using formula 1.

$SENTIMENT^{(i)}={N_{positive}^{(i)}-N_{negative}^{(i)}\over
N_{positive}^{(i)}+N_{negative}^{(i)}}$ (1)

Where $N_{positive}^{(i)}$ is the number of words which is in content script
at date $i$ and also a KOSELF positive word, and $N_{negative}^{(i)}$ is the
number of words which is in content script at date $i$ and also a KOSELF
negative word. We count duplicate words separately. For separating the
positively mentioned stocks and negatively mentioned stocks, we match
$SENTIMENT^{(i)}$ to each stock $j$ that is mentioned in content script at
date $i$.

Our apporach for evaluating the information content of the 3PROTV scripts are
in two main ways. One is analyzing its stock selection information, and the
other is its market timing. For investigating the possibility of information
content on stock selection, we deploy the standard event study methodology,
[8] but with some specific settings. One of those differences is that we use
state-of-the-art expected return model, the Fama-French 5 factor model, which
takes a form like formula. [6]

$R_{j}(t)-r_{f}(t)=\alpha_{j}+\beta_{j}\cdot RMRF(t)+s_{j}\cdot
SMB(t)+h_{j}\cdot HML(t)+r_{j}\cdot RMW(t)+c_{j}\cdot CMA(t)+\epsilon_{j}(t)$
(2)

Where $R_{j}(t)$ is the daily investment return of stock $j$ at day $t$,
$r_{f}(t)$ is the risk-free asset’s daily investment return at day $t$,
$RMRF(t)$ is the market portfolio’s daily excess return at day $t$, $SMB(t)$
is a daily return of zero-investment portfolio (or a long-short portfolio) of
small firm portfolio minus big firm portfolio at day $t$, $HML(t)$ is a daily
return of zero-investment portfolio of high value (high book-to-market ratio)
firm portfolio minus low value (low book-to-market ratio) firm portfolio at
day $t$, $RMW(t)$ is a daily return of zero-investment portfolio of robust
(high operating income-book value ratio) firm portfolio minus weak (low
operating income-book value ratio) firm portfolio at day $t$, $CMA(t)$ is a
daily return of zero-investment portfolio of conservative (low asset growth)
firm portfolio minus aggressive (high asset growth) firm portfolio at day $t$,
and $\epsilon_{j}(t)$ is a idiosyncratic variation of stock return. We
followed Fama and French (2015)’s method as same as possible while applying it
on the Korean stock market data gathered from DataGuide, with some adaption
like using KOSPI breakpoints to sort the stocks in KOSPI and KOSDAQ, which
correspond to using NYSE breakpoints in the original paper. Also, we extend
the model to have 1 leading and 1 lagging variables for each factors to
consider delayed or advance reaction, totaling up to 15 independent variables
for the expected return estimation.

Our event study methodology is as follows. First, based on relative date $0$
when the stock $j$ is mentioned by 3PROTV’s panel, we set a estimation period
from $-273$ to $-21$ for estimating Fama-French 5 factor model’s exposures
(i.e. the slopes like $\beta_{j},s_{j},\cdots$) and intercept $\alpha_{j}$
parameters for each stock. Then for $-20$ to $20$, the expected daily return
$\overline{R_{j}}(t)$ of the stock is calculated. We then derive abnormal
return $AR_{j}(t)$, which is nothing but

$AR_{j}(t)=R_{j}(t)-\overline{R_{j}}(t)$ (3)

We then average the $AR_{j}(t)$ by stocks by certain list of stocks $J$ to
obtain the average abnormal return $AAR_{j}(t)$, i.e.

$AAR_{j}(t)={1\over N}\sum_{j\in J}AR_{j}(t)$ (4)

where $N$ is the total number of stocks in $J$. Finally, we sum the
$AAR_{j}(t)$ over the $T$ days in the event window or sub-window
$[t_{0},t_{0}+T]$ to form the cumulative average abnormal return
$CAAR_{j}(t_{0}+T)$

$CAAR_{j}(t_{0},t_{0}+T)=\sum_{t=t_{0}}^{t_{0}+T}AAR_{j}(t)$ (5)

We set $t_{0}=-20$ and $T=40$ for full event window, with additional analysis
using $t_{0}=0$ and $T=20$ for event window period where the investor is
notified by 3PROTV. Then we compare pattern of CAAR between $J^{(+)}$ and
$J^{(-)}$, where $J^{(+)}$ consist of stock with corresponding
$SENTIMENT^{(i)}$ being positive, and $J^{(-)}$ being negative. By this rough
analysis, we get a sense of average stock price change patterns of which
3PROTV had flagged positive or negative. Comparing the two CAARs time series
will show whether 3PROTV’s information is already public (whether or not there
is a price reaction before the mention), or the accuracy of 3PROTV’s sentiment
prediction (positively flagged stock should outperform negatively flagged
stocks). Sample size of postively mentioned stocks are total 13, while
negatively mentioned stocks are total 54. The asymmetry is expected, due to
the general negative sentiment of 2022’s stock market investors.

For showing evidence for market timing information content, we develop a
preprocessed sentiment index $S(N,t)$ from $SENTIMENT^{(i)}$, which we
demonstrate that it has predictive power for sign of 2day leading market
portfolio return. We define a sentiment index $S(N,t)$ with a binary value
that is defined as equation 6.

$S(N,t)=\begin{cases}1&\text{if }SENTIMENT^{(t)}-{1\over
N}\sum_{i=t-N-1}^{t}SENTIMENT^{(i)}>0\\\ 0&\text{else}\\\ \end{cases}$ (6)

We report $R^{2}$ between $S(N,t-2)$ and $RMRF(t)$ with several different $N$s
for robustness check. Then we check the profitability of market timing using
$S(N,t-2)$. Lastly we consider the logit regression with $S(N,t-2)$ as its
independent variable and the sign of $RMRF(t)$ as its dependent variable. We
then control for change in news sentiment index (NSI) [9] whose daily values
are announced by the Bank of Korea, with also business cycle variable like
short interest rate and liquidity risk factor ($PctZero$). Interest rate for
91day Monetary Stabilization Bond is used as short interest rate and $PctZero$
is a daily ratio of zero-return stock within the market. Controlling for
business cycle variable is motivated by Sibley, Wang, Xing and Zhang (2016),
who suggested that baker-Wurgler sentiment index have cross-sectional return
prediction ability mainly due to the information content related to business
cycle variable. [10]

##  3 Empirical Results

###  3.1 Information Content on Stock Selection

Figure 1 and Figure 2 show the result of the first analysis using CCAR. We can
found that stocks in $J^{(+)}$ outperforms in average than stocks in $J^{(-)}$
within a event window, which means 3PROTV’s content sentiment roughly have
information content on the average stock performance. However comparing the
two figures, it seems that positive content sentiment information is already
being incorporated into the stock price before 12 days, with spike in stock
price right before the announcement. This implies that 3PROTV’s positively
flagged stocks and their information are likely to be already known to the
public.

Figure 1: Time series plot of cumulative average abnormal return of two
distinct stock group, the blue curve being a negatively flagged stock by
3PROTV, and orange curve being a positivly flagged stock by 3PROTV. The
vertical line is the announcement (mention) time, which is relatively zero. If
the announcement was not a trading day, we move the announcement date to the
nearlest future trading day.

Information with negative sentiment seems to have more new information
content, which price shock happening around the announcement date. The
lingering shock throughout the post-announcement period resembles the
investor’s disposition effect[7], as negative price shock discourages
investors to sell their stocks, forming an underreaction.

Figure 2: Time series plot of cumulative average abnormal return of two
distinct stock group, the blue curve being a negatively flagged stock by
3PROTV, and orange curve being a positivly flagged stock by 3PROTV. We limit
the event window The vertical line is the announcement (mention) time, which
is relatively zero. If the announcement was not a trading day, we move the
announcement date to the nearlest future trading day.

Long story short, 3PROTV’s content sentiment on stocks have independent
information content only on stock selection information with negative
sentiment. Information with positive sentiment have little information
content, with limitation to investment decision. Investors in these stocks
faces overreacted and volatile stock price.

###  3.2 Information Content on Market Timing

Considering the former result, it could be suspected that the script sentiment
can be used in other application. We focus on short-term market return
prediction.

We had mentioned the formula for $S(N,t)$, which is one when the current
sentiment of 3PROTV’s script is higher than its N moving average, and zero if
it is not. This methodology maximize the predictability in the sample, as it
is shown in Figure 3 . from $N$ ranging from 5 to 20, 10 out of 16 index at
$t-2$ show more than 1.5% of $R^{2}$, which is quite a high bar for market
portfolio return prediction model. We choose $t-2$ because the youtube content
could be released after the market is closed, making it impossible to trade
stocks using index at $t-1$. Some might argue that the performance of $S(N,t)$
is due to data snooping, not a real discovery. While it is true that $S(N,t)$
is not tested for out-of-sample data (which will be left for follow-up study),
we dispute that argument with some key point. One of two points is that
$S(N,t)$ have less information than $SENTIMENT^{(i)}$, which works as a
penalty rather than advantage. The other point is that at least the index is
robust to change in $N$, which the deployment of moving average is less data
mining than it seems. All things aside, it is reasonable to have a healthy
scepticism about our methodology, and replication study by using data after
2022 is required for further application.

Figure 3: $R^{2}$ between $S(N,t-2)$ and $RMRF(t)$ with difference. The dotted
line is $R^{2}=1.5\%$, which is the baseline performance for usable signal.
$R^{2}$ is the square of correlation between $S(N,t-2)$ and $RMRF(t)$. We
report that for every $N$, the correlation is positive.

We apply the sentiment index to trading using the ”Sentiment Strategy”, where
the trader long $RMRF(t)$ when $S(N,t-2)$ is 1 and short when $S(N,t-2)$ is 0.
We tested with setting the parameters to $N=5,10,20$, which in return show
outperformance ranging from 11.5% (N=5) to 23.6% (N=10) within 3 months.

Figure 4: Time series plot of sentiment strategy investment return and
investment return of $RMRF$.

###  3.3 Decomposing the Information Content on Market Timing

Lastly, we report the Logit regression result to show our attempts to
decompose the sentiment index’s information content on market timing. The
result is shown in Table 1, which is little bit surprising. Even controlling
lead variables of liquidity factor, interest rate and news sentiment change,
$S(10,t-2)$ still show strong predictive power. We fail to decompose the
sentiment index’s predictive power in few specific ways. One part is that
contradicting to theory of investor sentiment, $S(10,t-2)$ show positive
correlation between future market ups and downs, unlike $\Delta NSI(t)$’s
negative correlation with the market which is consistent with the theory. The
other part is that $PctZero(t)$ and $R_{f}(t)$ failed to absorb $S(10,t-2)$’s
predictive power, which could be due to the high-frequency nature of this
analysis. We conduct the analysis with linear regression, and while we do not
report the specifics, the result is fairly similar, with coefficient’s
statistical significance strengthen for the control variables.

| $S(10,t-2)$ | $\Delta NSI(t)$ | $PctZero(t)$ | $R_{f}(t)$
---|---|---|---|---
| 0.52 |  |  |
| (2.02) |  |  |
Logit Regression1 |  |  |  |
|  | -3.11 |  |
|  | (-2.23) |  |
Logit Regression2 |  |  |  |
|  |  | 27.55 | 0.66
|  |  | (1.47) | (0.85)
Logit Regression3 |  |  |  |
| 0.60 | -3.45 |  |
| (2.19) | (-2.35) |  |
Logit Regression4 |  |  |  |
| 0.77 | -4.00 | 46.04 | 0.96
Logit Regression5 | (2.48) | (-2.56) | (1.97) | (1.08)
Table 1: Five logistic regression result (regression slope values) that
regress stock market’s up(1) and down(0) by $S(10,t-2)$, $\Delta NSI(t)$ (the
change of NSI, $PctZero(t)$, and short interest rate $R_{f}(t)$. Values in
parentheses is the T-values that correspond to each slope above it. For each
regression, we exclude or include some variables to compare the $S(10,t-2)$’s
predictability when other variables are controlled.

Although we may not found the exact source of information content of $S(N,t)$,
we can report on the following point. One is that, in this sample period, the
$S(N,t)$, which was derived from 3PROTV, have some information content on
market timing, and the information content is somewhat independent from news
sentiment. We do not know $S(N,t)$ is a investor sentiment, however the
regression result suggets otherwise. $S(N,t)$’s information content do not
overlap with future $NSI$, which is the currently the best high frequency
proxy for investor sentiment available for the public. The positive sign of
the coefficent of $S(10,t-2)$ also show that the index is less likely to be a
sentiment index, and rather some kind of fundamental information indicator.
The other point is that the economic effect of $S(N,t)$ is quite strong and
robust. Even for the worst parameter choice, it yield over 10% excess return
than the buy-and-hold market portfolio. However the tests for validity of
$S(N,t)$ is quite limited at this moment. With lack of long-term historcal
data, currently we cannot concretely conclude for all time period that 3PROTV
holds information content on market timing.

However within our sample, we can carefully conclude that we found some
potentials that 3PROTV could have information content on market timing.

##  4 Conclusion

Due to the lack of data, it is unwise to result in sweeping conclusion, but we
discuss the result with a limited generality while focusing on its implication
to investment strategy. First, we found that 3PROTV holds information content
on stock selection, which means trading according to its information could
enable outperformance. However, most of the outperformance is due to delayed
reaction to negative information, and trading according to positive
information will cause higher volatility and market timing disadvantages.
Second, change in 3PROTV’s sentimental tone have predictive power of future
market return, implying trading according to 3PROTV does have some market
timing advantage in the total market sense.

It is interesting to point out that the content sentiment is more useful
regarding overall market investment rather than stock selection. We suspect
that this is because of the characteristics of general audience base of
3PROTV. As we discussed before, 3PROTV’s main target customer is individual
investor, not professionals or institutional investors. These individual
investors are free from index-tracking obligations. Also it is unlikely for
the audience of financial advisory YouTube channel to be satisfied with
investment performance of index-tracking funds, thus the audience are more
likely to incorporate the information about specific firms in the market, not
about the total market. Based on this hypothesis, we conjecture that the
individual stock-related information announced by 3PROTV is already
incorporated into the price before the annoucement, but overall market-related
information is not quickely incorporated into the price, leaving out the
opportunity of arbitrage.

That being said, we also note that this result cannot be stretched to other
areas, like generalizing about information content of all financial YouTube
channels or even within 3PROTV’s out-of-sample period. Also some might suggest
that natrual language processing of the scripts were quite rough and loose
(e.g. $SENTIMENT^{(i)}$ for stocks are more content sentiment rather than
individual stock sentiment), making the result of stock selection analysis
less reliable. Though we argue that this paper have contribution to the
literature, like the test for information content on emerging market social
media with the main contribution to the literature being not on stock
selection analysis but on analyzing YouTube contents about their information
contents on market timing, viewing financial YouTube contents in a time series
perspective, whose value is not well recognised.

## References

  * [1] Eric Abrahamson, Eli Amir, et al.  The information content of the president’s letter to shareholders.  Journal of Business Finance and Accounting, 23:1157–1182, 1996.
  * [2] Paul Asquith, Michael B Mikhail, and Andrea S Au.  Information content of equity analyst reports.  Journal of financial economics, 75(2):245–282, 2005.
  * [3] Malcolm Baker and Jeffrey Wurgler.  Investor sentiment in the stock market.  Journal of economic perspectives, 21(2):129–151, 2007.
  * [4] William H Beaver.  The information content of annual earnings announcements.  Journal of accounting research, pages 67–92, 1968.
  * [5] Suji Cho, Heung-Ku Kim, and Cheol Won Yang.  Building the korean sentiment lexicon for finance (koself).  Korean Journal of Financial Studies, 50(2):135–170, 2021.
  * [6] Eugene F Fama and Kenneth R French.  A five-factor asset pricing model.  Journal of financial economics, 116(1):1–22, 2015.
  * [7] Mark Grinblatt and Bing Han.  The disposition effect and momentum, 2002.
  * [8] Pamela P Peterson.  Event studies: A review of issues and methodology.  Quarterly journal of business and economics, pages 36–66, 1989.
  * [9] Beomseok Seo.  Machine-learning-based news sentiment index (nsi) of korea.  Bank of Korea WP, 15, 2022.
  * [10] Steven E Sibley, Yanchu Wang, Yuhang Xing, and Xiaoyan Zhang.  The information content of the sentiment index.  Journal of Banking & Finance, 62:164–179, 2016.
  * [11] Hong Kee Sul, Alan R Dennis, and Lingyao Yuan.  Trading on twitter: Using social media sentiment to predict stock returns.  Decision Sciences, 48(3):454–488, 2017.
  * [12] Hongkee Sul, Alan R Dennis, and Lingyao Ivy Yuan.  Trading on twitter: The financial information content of emotion in social media.  In 2014 47th Hawaii International Conference on System Sciences, pages 806–815. IEEE, 2014.