|
import streamlit as st
|
|
|
|
|
|
st.markdown("""
|
|
<style>
|
|
.main-title {
|
|
font-size: 36px;
|
|
color: #4A90E2;
|
|
font-weight: bold;
|
|
text-align: center;
|
|
}
|
|
.sub-title {
|
|
font-size: 24px;
|
|
color: #4A90E2;
|
|
margin-top: 20px;
|
|
}
|
|
.section {
|
|
background-color: #f9f9f9;
|
|
padding: 15px;
|
|
border-radius: 10px;
|
|
margin-top: 20px;
|
|
}
|
|
.section h2 {
|
|
font-size: 22px;
|
|
color: #4A90E2;
|
|
}
|
|
.section p, .section ul {
|
|
color: #666666;
|
|
}
|
|
.link {
|
|
color: #4A90E2;
|
|
text-decoration: none;
|
|
}
|
|
.benchmark-table {
|
|
width: 100%;
|
|
border-collapse: collapse;
|
|
margin-top: 20px;
|
|
}
|
|
.benchmark-table th, .benchmark-table td {
|
|
border: 1px solid #ddd;
|
|
padding: 8px;
|
|
text-align: left;
|
|
}
|
|
.benchmark-table th {
|
|
background-color: #4A90E2;
|
|
color: white;
|
|
}
|
|
.benchmark-table td {
|
|
background-color: #f2f2f2;
|
|
}
|
|
</style>
|
|
""", unsafe_allow_html=True)
|
|
|
|
|
|
st.markdown('<div class="main-title">Introduction to BERT Annotators in Spark NLP</div>', unsafe_allow_html=True)
|
|
st.markdown("""
|
|
<div class="section">
|
|
<p>Spark NLP provides a range of BERT-based annotators that leverage the power of Bidirectional Encoder Representations from Transformers (BERT) for various natural language processing tasks. These annotators are designed to deliver high performance and scalability in production environments. Below, we provide a detailed overview of four key BERT-based annotators available in Spark NLP:</p>
|
|
</div>
|
|
""", unsafe_allow_html=True)
|
|
st.write("")
|
|
|
|
tab1, tab2, tab3, tab4 = st.tabs(["BERT for Token Classification", "BERT for Zero-Shot Classification", "BERT for Sequence Classification", "BERT for Question Answering"])
|
|
|
|
with tab1:
|
|
st.markdown("""
|
|
<div class="section">
|
|
<h2>BERT for Token Classification</h2>
|
|
<p>The <strong>BertForTokenClassification</strong> annotator is fine-tuned for Named Entity Recognition (NER) tasks. Token classification involves labeling tokens, which are the smallest units of meaning in a text, with tags that represent specific entities. This process is crucial for understanding and extracting valuable information from text data. By identifying entities like names of people, organizations, locations, and more, token classification enables a wide range of applications, including:</p>
|
|
<ul>
|
|
<li><strong>Information Extraction:</strong> Automatically pulling out important information from large volumes of text.</li>
|
|
<li><strong>Document Categorization:</strong> Enhancing the organization and retrieval of documents based on identified entities.</li>
|
|
<li><strong>Improved Search Engine Relevancy:</strong> Enabling more accurate and context-aware search results.</li>
|
|
</ul>
|
|
<p>This annotator is highly effective for applications requiring precise entity recognition, ensuring that the identified entities are accurate and contextually relevant.</p>
|
|
<table class="benchmark-table">
|
|
<tr>
|
|
<th>Entity</th>
|
|
<th>Label</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Apple</td>
|
|
<td>ORGANIZATION</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Steve Jobs</td>
|
|
<td>PERSON</td>
|
|
</tr>
|
|
<tr>
|
|
<td>California</td>
|
|
<td>LOCATION</td>
|
|
</tr>
|
|
</table>
|
|
</div>
|
|
""", unsafe_allow_html=True)
|
|
|
|
|
|
st.markdown('<div class="sub-title">BERT Token Classification - NER CoNLL</div>', unsafe_allow_html=True)
|
|
st.markdown("""
|
|
<div class="section">
|
|
<p>The <strong>bert_base_token_classifier_conll03</strong> is a fine-tuned BERT model ready to use for Named Entity Recognition (NER) tasks. This model recognizes four types of entities: location (LOC), organizations (ORG), person (PER), and Miscellaneous (MISC).</p>
|
|
</div>
|
|
""", unsafe_allow_html=True)
|
|
|
|
|
|
st.markdown('<div class="sub-title">How to Use the Model</div>', unsafe_allow_html=True)
|
|
st.code('''
|
|
from sparknlp.base import *
|
|
from sparknlp.annotator import *
|
|
from pyspark.ml import Pipeline
|
|
from pyspark.sql.functions import col, expr
|
|
|
|
document_assembler = DocumentAssembler() \\
|
|
.setInputCol('text') \\
|
|
.setOutputCol('document')
|
|
|
|
sentence_detector = SentenceDetector() \\
|
|
.setInputCols(['document']) \\
|
|
.setOutputCol('sentence')
|
|
|
|
tokenizer = Tokenizer() \\
|
|
.setInputCols(['sentence']) \\
|
|
.setOutputCol('token')
|
|
|
|
tokenClassifier = BertForTokenClassification \\
|
|
.pretrained('bert_base_token_classifier_conll03', 'en') \\
|
|
.setInputCols(['token', 'sentence']) \\
|
|
.setOutputCol('ner') \\
|
|
.setCaseSensitive(True) \\
|
|
.setMaxSentenceLength(512)
|
|
|
|
ner_converter = NerConverter() \\
|
|
.setInputCols(['sentence', 'token', 'ner']) \\
|
|
.setOutputCol('entities')
|
|
|
|
pipeline = Pipeline(stages=[
|
|
document_assembler,
|
|
sentence_detector,
|
|
tokenizer,
|
|
tokenClassifier,
|
|
ner_converter
|
|
])
|
|
|
|
example = spark.createDataFrame([["""Apple Inc. is planning to open a new headquarters in Cupertino, California. The CEO, Tim Cook, announced this during the company's annual event on March 25th, 2023. Barack Obama, the 44th President of the United States, was born on August 4th, 1961, in Honolulu, Hawaii. He attended Harvard Law School and later became a community organizer in Chicago. Amazon reported a net revenue of $125.6 billion in Q4 of 2022, an increase of 9% compared to the previous year. Jeff Bezos, the founder of Amazon, mentioned that the company's growth in cloud computing has significantly contributed to this rise. Paris, the capital city of France, is renowned for its art, fashion, and culture. Key attractions include the Eiffel Tower, the Louvre Museum, and the Notre-Dame Cathedral. Visitors often enjoy a stroll along the Seine River and dining at local bistros. The study, conducted at the Mayo Clinic in Rochester, Minnesota, examined the effects of a new drug on patients with Type 2 diabetes. Results showed a significant reduction in blood sugar levels over a 12-month period. Serena Williams won her 24th Grand Slam title at the Wimbledon Championships in London, England. She defeated Naomi Osaka in a thrilling final match on July 13th, 2023. Google's latest smartphone, the Pixel 6, was unveiled at an event in New York City. Sundar Pichai, the CEO of Google, highlighted the phone's advanced AI capabilities and improved camera features. The Declaration of Independence was signed on July 4th, 1776, in Philadelphia, Pennsylvania. Thomas Jefferson, Benjamin Franklin, and John Adams were among the key figures who drafted this historic document."""]]).toDF("text")
|
|
result = pipeline.fit(example).transform(example)
|
|
|
|
result.select(
|
|
expr("explode(entities) as ner_chunk")
|
|
).select(
|
|
col("ner_chunk.result").alias("chunk"),
|
|
col("ner_chunk.metadata.entity").alias("ner_label")
|
|
).show(truncate=False)
|
|
''', language='python')
|
|
|
|
st.text("""
|
|
+--------------------+---------+
|
|
|chunk |ner_label|
|
|
+--------------------+---------+
|
|
|Apple Inc. |ORG |
|
|
|Cupertino |LOC |
|
|
|California |LOC |
|
|
|Tim Cook |PER |
|
|
|Barack Obama |PER |
|
|
|United States |LOC |
|
|
|Honolulu |LOC |
|
|
|Hawaii |LOC |
|
|
|Harvard Law School |ORG |
|
|
|Chicago |LOC |
|
|
|Amazon |ORG |
|
|
|Jeff Bezos |PER |
|
|
|Amazon |ORG |
|
|
|Paris |LOC |
|
|
|France |LOC |
|
|
|Eiffel Tower |LOC |
|
|
|Louvre Museum |LOC |
|
|
|Notre-Dame Cathedral|LOC |
|
|
|Seine River |LOC |
|
|
|Mayo Clinic |ORG |
|
|
+--------------------+---------+
|
|
""")
|
|
|
|
|
|
st.markdown('<div class="sub-title">Model Information</div>', unsafe_allow_html=True)
|
|
st.markdown("""
|
|
<table class="benchmark-table">
|
|
<tr>
|
|
<th>Attribute</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Model Name</strong></td>
|
|
<td>bert_base_token_classifier_conll03</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Compatibility</strong></td>
|
|
<td>Spark NLP 3.2.0+</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>License</strong></td>
|
|
<td>Open Source</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Edition</strong></td>
|
|
<td>Official</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Input Labels</strong></td>
|
|
<td>[token, document]</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Output Labels</strong></td>
|
|
<td>[ner]</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Language</strong></td>
|
|
<td>en</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Size</strong></td>
|
|
<td>404.3 MB</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Case sensitive</strong></td>
|
|
<td>true</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Max sentence length</strong></td>
|
|
<td>512</td>
|
|
</tr>
|
|
</table>
|
|
""", unsafe_allow_html=True)
|
|
|
|
|
|
st.markdown('<div class="sub-title">References</div>', unsafe_allow_html=True)
|
|
st.markdown("""
|
|
<div class="section">
|
|
<ul>
|
|
<li><a class="link" href="https://github.com/google-research/bert" target="_blank" rel="noopener">Google Research BERT</a></li>
|
|
<li><a class="link" href="https://arxiv.org/abs/1810.04805" target="_blank" rel="noopener">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a></li>
|
|
<li><a class="link" href="https://huggingface.co/bert-base-uncased" target="_blank" rel="noopener">Hugging Face BERT Models</a></li>
|
|
</ul>
|
|
</div>
|
|
""", unsafe_allow_html=True)
|
|
|
|
with tab2:
|
|
st.markdown("""
|
|
<div class="section">
|
|
<h2>BERT for Zero-Shot Classification</h2>
|
|
<p>The <strong>BertForZeroShotClassification</strong> annotator is designed to classify text into labels it has not seen during training. This is achieved using natural language inference (NLI) to determine the relationship between input text and potential labels. This capability is essential for applications where predefined categories are either unavailable or frequently change. Zero-shot classification is particularly useful for:</p>
|
|
<ul>
|
|
<li><strong>Dynamic Content Tagging:</strong> Automatically categorizing content without the need for a pre-existing label set.</li>
|
|
<li><strong>Sentiment Analysis:</strong> Analyzing sentiment for new and emerging topics without retraining the model.</li>
|
|
</ul>
|
|
<p>By leveraging this annotator, you can ensure flexibility and adaptability in text classification tasks, making it suitable for ever-changing data environments.</p>
|
|
<table class="benchmark-table">
|
|
<tr>
|
|
<th>Text</th>
|
|
<th>Predicted Category</th>
|
|
</tr>
|
|
<tr>
|
|
<td>"The new iPhone has amazing features"</td>
|
|
<td>Technology</td>
|
|
</tr>
|
|
<tr>
|
|
<td>"The economic growth has been significant this year"</td>
|
|
<td>Finance</td>
|
|
</tr>
|
|
</table>
|
|
</div>
|
|
""", unsafe_allow_html=True)
|
|
|
|
|
|
st.markdown('<div class="sub-title">BERT Zero-Shot Classification Base - MNLI</div>', unsafe_allow_html=True)
|
|
st.markdown("""
|
|
<div class="section">
|
|
<p>The <strong>bert_zero_shot_classifier_mnli</strong> model is designed for zero-shot text classification, making it suitable for scenarios where predefined categories are not available or frequently change. This model is fine-tuned on the MNLI dataset and leverages natural language inference (NLI) to determine relationships between input text and candidate labels. It allows for dynamic classification without a fixed number of classes, providing flexibility and adaptability for various applications.</p>
|
|
</div>
|
|
""", unsafe_allow_html=True)
|
|
|
|
|
|
st.markdown('<div class="sub-title">How to Use the Model</div>', unsafe_allow_html=True)
|
|
st.code('''
|
|
from sparknlp.base import *
|
|
from sparknlp.annotator import *
|
|
from pyspark.ml import Pipeline
|
|
|
|
document_assembler = DocumentAssembler() \\
|
|
.setInputCol('text') \\
|
|
.setOutputCol('document')
|
|
|
|
tokenizer = Tokenizer() \\
|
|
.setInputCols(['document']) \\
|
|
.setOutputCol('token')
|
|
|
|
|
|
zeroShotClassifier = BertForZeroShotClassification \\
|
|
.pretrained('bert_zero_shot_classifier_mnli', 'xx') \\
|
|
.setInputCols(['token', 'document']) \\
|
|
.setOutputCol('class') \\
|
|
.setCaseSensitive(True) \\
|
|
.setMaxSentenceLength(512) \\
|
|
.setCandidateLabels(["urgent", "mobile", "travel", "movie", "music", "sport", "weather", "technology"])
|
|
|
|
pipeline = Pipeline(stages=[
|
|
document_assembler,
|
|
tokenizer,
|
|
zeroShotClassifier
|
|
])
|
|
|
|
example = spark.createDataFrame([['In today’s world, staying updated with urgent information is crucial as events can unfold rapidly and require immediate attention.']]).toDF("text")
|
|
result = pipeline.fit(example).transform(example)
|
|
|
|
result.select('document.result', 'class.result').show(truncate=False)
|
|
''', language='python')
|
|
|
|
st.text("""
|
|
+------------------------------------------------------------------------------------------------------------------------------------+--------+
|
|
|result |result |
|
|
+------------------------------------------------------------------------------------------------------------------------------------+--------+
|
|
|[In today’s world, staying updated with urgent information is crucial as events can unfold rapidly and require immediate attention.]|[urgent]|
|
|
+------------------------------------------------------------------------------------------------------------------------------------+--------+
|
|
""")
|
|
|
|
|
|
st.markdown('<div class="sub-title">Model Information</div>', unsafe_allow_html=True)
|
|
st.markdown("""
|
|
<table class="benchmark-table">
|
|
<tr>
|
|
<th>Attribute</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Model Name</strong></td>
|
|
<td>bert_zero_shot_classifier_mnli</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Compatibility</strong></td>
|
|
<td>Spark NLP 5.2.4+</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>License</strong></td>
|
|
<td>Open Source</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Edition</strong></td>
|
|
<td>Official</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Input Labels</strong></td>
|
|
<td>[token, document]</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Output Labels</strong></td>
|
|
<td>[label]</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Language</strong></td>
|
|
<td>xx</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Size</strong></td>
|
|
<td>409.1 MB</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Case sensitive</strong></td>
|
|
<td>true</td>
|
|
</tr>
|
|
</table>
|
|
""", unsafe_allow_html=True)
|
|
|
|
|
|
st.markdown('<div class="sub-title">References</div>', unsafe_allow_html=True)
|
|
st.markdown("""
|
|
<div class="section">
|
|
<ul>
|
|
<li><a class="link" href="https://github.com/google-research/bert" target="_blank" rel="noopener">Google Research BERT</a></li>
|
|
<li><a class="link" href="https://arxiv.org/abs/1810.04805" target="_blank" rel="noopener">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a></li>
|
|
<li><a class="link" href="https://huggingface.co/bert-base-uncased" target="_blank" rel="noopener">Hugging Face BERT Models</a></li>
|
|
</ul>
|
|
</div>
|
|
""", unsafe_allow_html=True)
|
|
|
|
with tab3:
|
|
st.markdown("""
|
|
<div class="section">
|
|
<h2>BERT for Sequence Classification</h2>
|
|
<p>The <strong>BertForSequenceClassification</strong> annotator is fine-tuned to classify entire sequences of text. This involves understanding the context of the entire sequence, which is crucial for tasks that require a holistic view of the input text. Sequence classification is highly effective for:</p>
|
|
<ul>
|
|
<li><strong>Sentiment Analysis:</strong> Determining the overall sentiment of a given piece of text.</li>
|
|
<li><strong>Spam Detection:</strong> Identifying unsolicited or irrelevant messages.</li>
|
|
<li><strong>Document Classification:</strong> Categorizing documents into predefined categories.</li>
|
|
</ul>
|
|
<p>With its ability to deliver accurate classification results, this annotator is widely used in various text analysis applications.</p>
|
|
<table class="benchmark-table">
|
|
<tr>
|
|
<th>Text</th>
|
|
<th>Predicted Sentiment</th>
|
|
</tr>
|
|
<tr>
|
|
<td>"I love this product, it's fantastic!"</td>
|
|
<td>Positive</td>
|
|
</tr>
|
|
<tr>
|
|
<td>"The service was terrible, I'm very disappointed."</td>
|
|
<td>Negative</td>
|
|
</tr>
|
|
</table>
|
|
</div>
|
|
""", unsafe_allow_html=True)
|
|
|
|
|
|
st.markdown('<div class="sub-title">English BertForSequenceClassification Cased model (from yonichi)</div>', unsafe_allow_html=True)
|
|
st.markdown("""
|
|
<div class="section">
|
|
<p>The <strong>bert_classifier_cbert</strong> model is a pretrained BertForSequenceClassification model. Adapted from Hugging Face and curated for scalability and production-readiness using Spark NLP, this model is designed for sequence classification tasks such as sentiment analysis. It is capable of classifying text into positive, negative, and neutral sentiments, providing valuable insights for various applications.</p>
|
|
</div>
|
|
""", unsafe_allow_html=True)
|
|
|
|
|
|
st.markdown('<div class="sub-title">How to Use the Model</div>', unsafe_allow_html=True)
|
|
st.code('''
|
|
from sparknlp.base import *
|
|
from sparknlp.annotator import *
|
|
from pyspark.ml import Pipeline
|
|
from pyspark.sql.functions import col, expr
|
|
|
|
# Document Assembler
|
|
document_assembler = DocumentAssembler() \\
|
|
.setInputCol('text') \\
|
|
.setOutputCol('document')
|
|
|
|
# Sentence Detector
|
|
sentence_detector = SentenceDetector() \\
|
|
.setInputCols(['document']) \\
|
|
.setOutputCol('sentence')
|
|
|
|
# Tokenizer
|
|
tokenizer = Tokenizer() \\
|
|
.setInputCols(['sentence']) \\
|
|
.setOutputCol('token')
|
|
|
|
# Sequence Classifier
|
|
sequence_classifier = BertForSequenceClassification.pretrained("bert_classifier_cbert", "en") \\
|
|
.setInputCols(['sentence', 'token']) \\
|
|
.setOutputCol('class')
|
|
|
|
# Pipeline
|
|
pipeline = Pipeline(stages=[
|
|
document_assembler,
|
|
sentence_detector,
|
|
tokenizer,
|
|
sequence_classifier
|
|
])
|
|
|
|
# Create example DataFrame
|
|
example = spark.createDataFrame([("Apple Inc. is planning to open a new headquarters in Cupertino, California. The CEO, Tim Cook, announced this during the company's annual event on March 25th, 2023. Barack Obama, the 44th President of the United States, was born on August 4th, 1961, in Honolulu, Hawaii. He attended Harvard Law School and later became a community organizer in Chicago. Amazon reported a net revenue of $125.6 billion in Q4 of 2022, an increase of 9% compared to the previous year. Jeff Bezos, the founder of Amazon, mentioned that the company's growth in cloud computing has significantly contributed to this rise. Paris, the capital city of France, is renowned for its art, fashion, and culture. Key attractions include the Eiffel Tower, the Louvre Museum, and the Notre-Dame Cathedral. Visitors often enjoy a stroll along the Seine River and dining at local bistros. The study, conducted at the Mayo Clinic in Rochester, Minnesota, examined the effects of a new drug on patients with Type 2 diabetes. Results showed a significant reduction in blood sugar levels over a 12-month period. Serena Williams won her 24th Grand Slam title at the Wimbledon Championships in London, England. She defeated Naomi Osaka in a thrilling final match on July 13th, 2023. Google's latest smartphone, the Pixel 6, was unveiled at an event in New York City. Sundar Pichai, the CEO of Google, highlighted the phone's advanced AI capabilities and improved camera features. The Declaration of Independence was signed on July 4th, 1776, in Philadelphia, Pennsylvania. Thomas Jefferson, Benjamin Franklin, and John Adams were among the key figures who drafted this historic document.",)], ["text"])
|
|
|
|
# Fit and transform the data
|
|
model = pipeline.fit(example)
|
|
result = model.transform(example)
|
|
|
|
from pyspark.sql.functions import col
|
|
|
|
# Show results in a structured format for sentence-based classification
|
|
result.select(
|
|
col('sentence.result').alias('sentences'),
|
|
col('class.result').alias('classifications')
|
|
).rdd.flatMap(lambda row: list(zip(row['sentences'], row['classifications']))).toDF(['sentence', 'classification']).show(truncate=False)
|
|
|
|
''', language='python')
|
|
|
|
st.text("""
|
|
+-------------------------------------------------------------------------------------------------------------------------------------+--------------+
|
|
|sentence |classification|
|
|
+-------------------------------------------------------------------------------------------------------------------------------------+--------------+
|
|
|Apple Inc. is planning to open a new headquarters in Cupertino, California. |Neutral |
|
|
|The CEO, Tim Cook, announced this during the company's annual event on March 25th, 2023. |Dovish |
|
|
|Barack Obama, the 44th President of the United States, was born on August 4th, 1961, in Honolulu, Hawaii. |Neutral |
|
|
|He attended Harvard Law School and later became a community organizer in Chicago. |Neutral |
|
|
|Amazon reported a net revenue of $125.6 billion in Q4 of 2022, an increase of 9% compared to the previous year. |Neutral |
|
|
|Jeff Bezos, the founder of Amazon, mentioned that the company's growth in cloud computing has significantly contributed to this rise.|Dovish |
|
|
|Paris, the capital city of France, is renowned for its art, fashion, and culture. |Neutral |
|
|
|Key attractions include the Eiffel Tower, the Louvre Museum, and the Notre-Dame Cathedral. |Neutral |
|
|
|Visitors often enjoy a stroll along the Seine River and dining at local bistros. |Neutral |
|
|
|The study, conducted at the Mayo Clinic in Rochester, Minnesota, examined the effects of a new drug on patients with Type 2 diabetes.|Dovish |
|
|
|Results showed a significant reduction in blood sugar levels over a 12-month period. |Neutral |
|
|
|Serena Williams won her 24th Grand Slam title at the Wimbledon Championships in London, England. |Hawkish |
|
|
|She defeated Naomi Osaka in a thrilling final match on July 13th, 2023. |Neutral |
|
|
|Google's latest smartphone, the Pixel 6, was unveiled at an event in New York City. |Dovish |
|
|
|Sundar Pichai, the CEO of Google, highlighted the phone's advanced AI capabilities and improved camera features. |Dovish |
|
|
|The Declaration of Independence was signed on July 4th, 1776, in Philadelphia, Pennsylvania. |Hawkish |
|
|
|Thomas Jefferson, Benjamin Franklin, and John Adams were among the key figures who drafted this historic document. |Neutral |
|
|
+-------------------------------------------------------------------------------------------------------------------------------------+--------------+
|
|
""")
|
|
|
|
|
|
st.markdown('<div class="sub-title">Model Information</div>', unsafe_allow_html=True)
|
|
st.markdown("""
|
|
<table class="benchmark-table">
|
|
<tr>
|
|
<th>Attribute</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Model Name</strong></td>
|
|
<td>bert_classifier_cbert</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Compatibility</strong></td>
|
|
<td>Spark NLP 4.2.0+</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>License</strong></td>
|
|
<td>Open Source</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Edition</strong></td>
|
|
<td>Official</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Input Labels</strong></td>
|
|
<td>[document, token]</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Output Labels</strong></td>
|
|
<td>[class]</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Language</strong></td>
|
|
<td>en</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Size</strong></td>
|
|
<td>412.2 MB</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Case sensitive</strong></td>
|
|
<td>true</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Max sentence length</strong></td>
|
|
<td>256</td>
|
|
</tr>
|
|
</table>
|
|
""", unsafe_allow_html=True)
|
|
|
|
|
|
st.markdown('<div class="sub-title">References</div>', unsafe_allow_html=True)
|
|
st.markdown("""
|
|
<div class="section">
|
|
<ul>
|
|
<li><a class="link" href="https://huggingface.co/yonichi/cbert" target="_blank" rel="noopener">Hugging Face cBERT Model</a></li>
|
|
</ul>
|
|
</div>
|
|
""", unsafe_allow_html=True)
|
|
|
|
with tab4:
|
|
st.markdown("""
|
|
<div class="section">
|
|
<h2>BERT for Question Answering</h2>
|
|
<p>The <strong>BertForQuestionAnswering</strong> annotator is fine-tuned to provide answers to questions based on a given context. This involves extracting relevant information from a passage of text in response to a specific query, making it ideal for applications requiring precise information retrieval. Question answering is particularly useful for:</p>
|
|
<ul>
|
|
<li><strong>Building Question-Answering Systems:</strong> Creating systems that can automatically answer user queries.</li>
|
|
<li><strong>Customer Support Bots:</strong> Providing accurate and timely responses to customer inquiries.</li>
|
|
<li><strong>Information Retrieval:</strong> Extracting specific information from large volumes of text.</li>
|
|
</ul>
|
|
<p>By leveraging this annotator, you can enhance the ability to extract and deliver accurate information from text data.</p>
|
|
<table class="benchmark-table">
|
|
<tr>
|
|
<th>Context</th>
|
|
<th>Question</th>
|
|
<th>Predicted Answer</th>
|
|
</tr>
|
|
<tr>
|
|
<td>"The Eiffel Tower is one of the most recognizable structures in the world. It was constructed in 1889 as the entrance arch to the 1889 World's Fair held in Paris, France."</td>
|
|
<td>"When was the Eiffel Tower constructed?"</td>
|
|
<td>1889</td>
|
|
</tr>
|
|
<tr>
|
|
<td>"The Amazon rainforest, also known as Amazonia, is a vast tropical rainforest in South America. It is home to an incredible diversity of flora and fauna."</td>
|
|
<td>"What is the Amazon rainforest also known as?"</td>
|
|
<td>Amazonia</td>
|
|
</tr>
|
|
</table>
|
|
</div>
|
|
""", unsafe_allow_html=True)
|
|
|
|
|
|
st.markdown('<div class="sub-title">bert_qa_large_uncased_whole_word_masking_finetuned_squad</div>', unsafe_allow_html=True)
|
|
st.markdown("""
|
|
<div class="section">
|
|
<p>This model is a pretrained BERT model, adapted from Hugging Face, curated to provide scalability and production-readiness using Spark NLP. It is designed to handle question-answering tasks effectively.</p>
|
|
</div>
|
|
""", unsafe_allow_html=True)
|
|
|
|
|
|
st.markdown('<div class="sub-title">How to Use the Model</div>', unsafe_allow_html=True)
|
|
st.code('''
|
|
from sparknlp.base import *
|
|
from sparknlp.annotator import *
|
|
from pyspark.ml import Pipeline
|
|
from pyspark.sql.functions import col, expr
|
|
|
|
# Document Assembler
|
|
document_assembler = MultiDocumentAssembler()\\
|
|
.setInputCols(["question", "context"]) \\
|
|
.setOutputCols(["document_question", "document_context"])
|
|
|
|
# BertForQuestionAnswering
|
|
spanClassifier = BertForQuestionAnswering.pretrained("bert_qa_large_uncased_whole_word_masking_finetuned_squad","en") \\
|
|
.setInputCols(["document_question", "document_context"]) \\
|
|
.setOutputCol("answer") \\
|
|
.setCaseSensitive(True)
|
|
|
|
# Pipeline
|
|
pipeline = Pipeline().setStages([
|
|
document_assembler,
|
|
spanClassifier
|
|
])
|
|
|
|
# Create example DataFrame
|
|
example = spark.createDataFrame([["What's my name?", "My name is Clara and I live in Berkeley."]]).toDF("question", "context")
|
|
|
|
# Fit and transform the data
|
|
model = pipeline.fit(example)
|
|
result = model.transform(example)
|
|
|
|
# Show results
|
|
result.select('document_question.result', 'answer.result').show(truncate=False)
|
|
''', language='python')
|
|
|
|
st.text("""
|
|
+-----------------+-------+
|
|
|result |result |
|
|
+-----------------+-------+
|
|
|[What's my name?]|[Clara]|
|
|
+-----------------+-------+
|
|
""")
|
|
|
|
|
|
st.markdown('<div class="sub-title">Model Information</div>', unsafe_allow_html=True)
|
|
st.markdown("""
|
|
<table class="benchmark-table">
|
|
<tr>
|
|
<th>Attribute</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Model Name</strong></td>
|
|
<td>bert_qa_large_uncased_whole_word_masking_finetuned_squad</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Compatibility</strong></td>
|
|
<td>Spark NLP 4.4.0+</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>License</strong></td>
|
|
<td>Open Source</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Edition</strong></td>
|
|
<td>Official</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Input Labels</strong></td>
|
|
<td>[document_question, document_context]</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Output Labels</strong></td>
|
|
<td>[answer]</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Language</strong></td>
|
|
<td>en</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Size</strong></td>
|
|
<td>1.3 GB</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Case sensitive</strong></td>
|
|
<td>false</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Max sentence length</strong></td>
|
|
<td>512</td>
|
|
</tr>
|
|
</table>
|
|
""", unsafe_allow_html=True)
|
|
|
|
|
|
st.markdown('<div class="sub-title">References</div>', unsafe_allow_html=True)
|
|
st.markdown("""
|
|
<div class="section">
|
|
<ul>
|
|
<li><a class="link" href="https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad" target="_blank" rel="noopener">BertForQuestionAnswering Model</a></li>
|
|
<li><a class="link" href="https://arxiv.org/abs/1810.04805" target="_blank" rel="noopener">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a></li>
|
|
<li><a class="link" href="https://github.com/google-research/bert" target="_blank" rel="noopener">Google Research BERT</a></li>
|
|
</ul>
|
|
</div>
|
|
""", unsafe_allow_html=True)
|
|
|
|
st.markdown("""
|
|
<div class="section">
|
|
<h2>Conclusion</h2>
|
|
<p>In this guide, we've explored a range of BERT-based annotators and models available in Spark NLP, each tailored to specific natural language processing tasks. Here's a summary of the four key BERT annotators and their respective models:</p>
|
|
<ul>
|
|
<li><strong>BERT for Sequence Classification</strong> - The <code>BertForSequenceClassification</code> annotator, exemplified by the <code>bert_classifier_cbert</code> model, excels in classifying entire sequences of text. This model is particularly useful for tasks like sentiment analysis, spam detection, and document categorization, providing insights into the overall sentiment of a text or its classification into predefined categories.</li>
|
|
<li><strong>BERT for Token Classification</strong> - The <code>BertForTokenClassification</code> annotator, with the <code>bert_base_token_classifier_conll03</code> model, specializes in Named Entity Recognition (NER). This annotator identifies entities such as people, organizations, locations, and more within a text, making it invaluable for information extraction and document categorization.</li>
|
|
<li><strong>BERT for Zero-Shot Classification</strong> - The <code>BertForZeroShotClassification</code> annotator, represented by the <code>bert_zero_shot_classifier_mnli</code> model, offers a flexible approach to text classification without requiring a predefined set of categories. It leverages natural language inference (NLI) to classify text into dynamically chosen labels, making it ideal for applications with evolving or unknown categories.</li>
|
|
<li><strong>BERT for Question Answering</strong> - The <code>BertForQuestionAnswering</code> annotator, though not highlighted in this specific instance, is designed to extract answers from a given context based on a query. This model is highly effective for building question-answering systems and customer support bots, facilitating precise information retrieval from large text corpora.</li>
|
|
</ul>
|
|
<p>Each of these models and annotators demonstrates the versatility and power of BERT-based approaches in natural language processing. Whether you need to classify sequences, identify entities, handle zero-shot classification, or answer questions, Spark NLP provides robust tools to enhance your text analysis capabilities. Leveraging these models allows for scalable and production-ready solutions in various applications, from sentiment analysis to dynamic content tagging.</p>
|
|
</div>
|
|
""", unsafe_allow_html=True)
|
|
|
|
|
|
st.markdown('<div class="sub-title">Community & Support</div>', unsafe_allow_html=True)
|
|
st.markdown("""
|
|
<div class="section">
|
|
<ul>
|
|
<li><a class="link" href="https://sparknlp.org/" target="_blank">Official Website</a>: Documentation and examples</li>
|
|
<li><a class="link" href="https://join.slack.com/t/spark-nlp/shared_invite/zt-198dipu77-L3UWNe_AJ8xqDk0ivmih5Q" target="_blank">Slack</a>: Live discussion with the community and team</li>
|
|
<li><a class="link" href="https://github.com/JohnSnowLabs/spark-nlp" target="_blank">GitHub</a>: Bug reports, feature requests, and contributions</li>
|
|
<li><a class="link" href="https://medium.com/spark-nlp" target="_blank">Medium</a>: Spark NLP articles</li>
|
|
<li><a class="link" href="https://www.youtube.com/channel/UCmFOjlpYEhxf_wJUDuz6xxQ/videos" target="_blank">YouTube</a>: Video tutorials</li>
|
|
</ul>
|
|
</div>
|
|
""", unsafe_allow_html=True) |
