What's a good machine learning tool for extracting keywords that's compatible with pyspark
https://github.com/MaartenGr/KeyBERT
GitHub - MaartenGr/KeyBERT: Minimal keyword extraction with BERT
2023-01-18
MaartenGr
<div><div>
 <article><p><a href="https://pypi.org/project/keybert/"></a>
<a href="https://github.com/MaartenGr/keybert/blob/master/LICENSE"></a>
<a href="https://pypi.org/project/keybert/"></a>
<a href="https://pypi.org/project/keybert/"></a>
<a href="https://colab.research.google.com/drive/1OxpgwKqSzODtO3vS7Xe1nEmZMCAIMckX?usp=sharing"></a></p>
<p><a href="/MaartenGr/KeyBERT/blob/master/images/logo.png"></a></p>
<h2>KeyBERT</h2>
<p>KeyBERT is a minimal and easy-to-use keyword extraction technique that leverages BERT embeddings to
create keywords and keyphrases that are most similar to a document.</p>
<p>Corresponding medium post can be found <a href="https://towardsdatascience.com/keyword-extraction-with-bert-724efca412ea">here</a>.</p>
<h2>Table of Contents</h2>
 
<ol>
<li><a href="#about">About the Project</a></li>
<li><a href="#gettingstarted">Getting Started</a><br />
2.1. <a href="#installation">Installation</a><br />
2.2. <a href="#usage">Basic Usage</a><br />
2.3. <a href="#maxsum">Max Sum Distance</a><br />
2.4. <a href="#maximal">Maximal Marginal Relevance</a><br />
2.5. <a href="#embeddings">Embedding Models</a></li>
</ol>
 
<h2>1. About the Project</h2>
<p><a href="#toc">Back to ToC</a></p>
<p>Although there are already many methods available for keyword generation
(e.g.,
<a href="https://github.com/aneesha/RAKE">Rake</a>,
<a href="https://github.com/LIAAD/yake">YAKE!</a>, TF-IDF, etc.)
I wanted to create a very basic, but powerful method for extracting keywords and keyphrases.
This is where <strong>KeyBERT</strong> comes in! Which uses BERT-embeddings and simple cosine similarity
to find the sub-phrases in a document that are the most similar to the document itself.</p>
<p>First, document embeddings are extracted with BERT to get a document-level representation.
Then, word embeddings are extracted for N-gram words/phrases. Finally, we use cosine similarity
to find the words/phrases that are the most similar to the document. The most similar words could
then be identified as the words that best describe the entire document.</p>
<p>KeyBERT is by no means unique and is created as a quick and easy method
for creating keywords and keyphrases. Although there are many great
papers and solutions out there that use BERT-embeddings
(e.g.,
<a href="https://github.com/pranav-ust/BERT-keyphrase-extraction">1</a>,
<a href="https://github.com/ibatra/BERT-Keyword-Extractor">2</a>,
<a href="https://www.preprints.org/manuscript/201908.0073/download/final_file">3</a>,
), I could not find a BERT-based solution that did not have to be trained from scratch and
could be used for beginners (<strong>correct me if I'm wrong!</strong>).
Thus, the goal was a <code>pip install keybert</code> and at most 3 lines of code in usage.</p>
<h2>2. Getting Started</h2>
<p><a href="#toc">Back to ToC</a></p>
<h3>2.1. Installation</h3>
<p>Installation can be done using <a href="https://pypi.org/project/keybert/">pypi</a>:</p>
<p>You may want to install more depending on the transformers and language backends that you will be using. The possible installations are:</p>
<div><pre><code>pip install keybert[flair]
pip install keybert[gensim]
pip install keybert[spacy]
pip install keybert[use]
</code></pre></div>
<h3>2.2. Usage</h3>
<p>The most minimal example can be seen below for the extraction of keywords:</p>
<div><pre>from keybert import KeyBERT
doc = """
 Supervised learning is the machine learning task of learning a function that
 maps an input to an output based on example input-output pairs. It infers a
 function from labeled training data consisting of a set of training examples.
 In supervised learning, each example is a pair consisting of an input object
 (typically a vector) and a desired output value (also called the supervisory signal).
 A supervised learning algorithm analyzes the training data and produces an inferred function,
 which can be used for mapping new examples. An optimal scenario will allow for the
 algorithm to correctly determine the class labels for unseen instances. This requires
 the learning algorithm to generalize from the training data to unseen situations in a
 'reasonable' way (see inductive bias).
 """
kw_model = KeyBERT()
keywords = kw_model.extract_keywords(doc)</pre></div>
<p>You can set <code>keyphrase_ngram_range</code> to set the length of the resulting keywords/keyphrases:</p>
<div><pre>&gt;&gt;&gt; kw_model.extract_keywords(doc, keyphrase_ngram_range=(1, 1), stop_words=None)
[('learning', 0.4604),
 ('algorithm', 0.4556),
 ('training', 0.4487),
 ('class', 0.4086),
 ('mapping', 0.3700)]</pre></div>
<p>To extract keyphrases, simply set <code>keyphrase_ngram_range</code> to (1, 2) or higher depending on the number
of words you would like in the resulting keyphrases:</p>
<div><pre>&gt;&gt;&gt; kw_model.extract_keywords(doc, keyphrase_ngram_range=(1, 2), stop_words=None)
[('learning algorithm', 0.6978),
 ('machine learning', 0.6305),
 ('supervised learning', 0.5985),
 ('algorithm analyzes', 0.5860),
 ('learning function', 0.5850)]</pre></div>
<p>We can highlight the keywords in the document by simply setting <code>highlight</code>:</p>
<div><pre>keywords = kw_model.extract_keywords(doc, highlight=True)</pre></div>
<p><a href="/MaartenGr/KeyBERT/blob/master/images/highlight.png"></a></p>
<p><strong>NOTE</strong>: For a full overview of all possible transformer models see <a href="https://www.sbert.net/docs/pretrained_models.html">sentence-transformer</a>.
I would advise either <code>"all-MiniLM-L6-v2"</code> for English documents or <code>"paraphrase-multilingual-MiniLM-L12-v2"</code>
for multi-lingual documents or any other language.</p>
<h3>2.3. Max Sum Distance</h3>
<p>To diversify the results, we take the 2 x top_n most similar words/phrases to the document.
Then, we take all top_n combinations from the 2 x top_n words and extract the combination
that are the least similar to each other by cosine similarity.</p>
<div><pre>&gt;&gt;&gt; kw_model.extract_keywords(doc, keyphrase_ngram_range=(3, 3), stop_words='english',
 use_maxsum=True, nr_candidates=20, top_n=5)
[('set training examples', 0.7504),
 ('generalize training data', 0.7727),
 ('requires learning algorithm', 0.5050),
 ('supervised learning algorithm', 0.3779),
 ('learning machine learning', 0.2891)]</pre></div>
<h3>2.4. Maximal Marginal Relevance</h3>
<p>To diversify the results, we can use Maximal Margin Relevance (MMR) to create
keywords / keyphrases which is also based on cosine similarity. The results
with <strong>high diversity</strong>:</p>
<div><pre>&gt;&gt;&gt; kw_model.extract_keywords(doc, keyphrase_ngram_range=(3, 3), stop_words='english',
 use_mmr=True, diversity=0.7)
[('algorithm generalize training', 0.7727),
 ('labels unseen instances', 0.1649),
 ('new examples optimal', 0.4185),
 ('determine class labels', 0.4774),
 ('supervised learning algorithm', 0.7502)]</pre></div>
<p>The results with <strong>low diversity</strong>:</p>
<div><pre>&gt;&gt;&gt; kw_model.extract_keywords(doc, keyphrase_ngram_range=(3, 3), stop_words='english',
 use_mmr=True, diversity=0.2)
[('algorithm generalize training', 0.7727),
 ('supervised learning algorithm', 0.7502),
 ('learning machine learning', 0.7577),
 ('learning algorithm analyzes', 0.7587),
 ('learning algorithm generalize', 0.7514)]</pre></div>
<h3>2.5. Embedding Models</h3>
<p>KeyBERT supports many embedding models that can be used to embed the documents and words:</p>
<ul>
<li>Sentence-Transformers</li>
<li>Flair</li>
<li>Spacy</li>
<li>Gensim</li>
<li>USE</li>
</ul>
<p>Click <a href="https://maartengr.github.io/KeyBERT/guides/embeddings.html">here</a> for a full overview of all supported embedding models.</p>
<p><strong>Sentence-Transformers</strong><br />
You can select any model from <code>sentence-transformers</code> <a href="https://www.sbert.net/docs/pretrained_models.html">here</a>
and pass it through KeyBERT with <code>model</code>:</p>
<div><pre>from keybert import KeyBERT
kw_model = KeyBERT(model='all-MiniLM-L6-v2')</pre></div>
<p>Or select a SentenceTransformer model with your own parameters:</p>
<div><pre>from keybert import KeyBERT
from sentence_transformers import SentenceTransformer
sentence_model = SentenceTransformer("all-MiniLM-L6-v2")
kw_model = KeyBERT(model=sentence_model)</pre></div>
<p><strong>Flair</strong><br />
<a href="https://github.com/flairNLP/flair">Flair</a> allows you to choose almost any embedding model that
is publicly available. Flair can be used as follows:</p>
<div><pre>from keybert import KeyBERT
from flair.embeddings import TransformerDocumentEmbeddings
roberta = TransformerDocumentEmbeddings('roberta-base')
kw_model = KeyBERT(model=roberta)</pre></div>
<p>You can select any 🤗 transformers model <a href="https://huggingface.co/models">here</a>.</p>
<h2>Citation</h2>
<p>To cite KeyBERT in your work, please use the following bibtex reference:</p>
<div><pre>@misc{grootendorst2020keybert,
 author = {Maarten Grootendorst},
 title = {KeyBERT: Minimal keyword extraction with BERT.},
 year = 2020,
 publisher = {Zenodo},
 version = {v0.3.0},
 doi = {10.5281/zenodo.4461265},
 url = {https://doi.org/10.5281/zenodo.4461265}
}</pre></div>
<h2>References</h2>
<p>Below, you can find several resources that were used for the creation of KeyBERT
but most importantly, these are amazing resources for creating impressive keyword extraction models:</p>
<p><strong>Papers</strong>:</p>
<ul>
<li>Sharma, P., &amp; Li, Y. (2019). <a href="https://www.preprints.org/manuscript/201908.0073/download/final_file">Self-Supervised Contextual Keyword and Keyphrase Retrieval with Self-Labelling.</a></li>
</ul>
<p><strong>Github Repos</strong>:</p>
<ul>
<li><a href="https://github.com/thunlp/BERT-KPE">https://github.com/thunlp/BERT-KPE</a></li>
<li><a href="https://github.com/ibatra/BERT-Keyword-Extractor">https://github.com/ibatra/BERT-Keyword-Extractor</a></li>
<li><a href="https://github.com/pranav-ust/BERT-keyphrase-extraction">https://github.com/pranav-ust/BERT-keyphrase-extraction</a></li>
<li><a href="https://github.com/swisscom/ai-research-keyphrase-extraction">https://github.com/swisscom/ai-research-keyphrase-extraction</a></li>
</ul>
<p><strong>MMR</strong>:
The selection of keywords/keyphrases was modeled after:</p>
<ul>
<li><a href="https://github.com/swisscom/ai-research-keyphrase-extraction">https://github.com/swisscom/ai-research-keyphrase-extraction</a></li>
</ul>
<p><strong>NOTE</strong>: If you find a paper or github repo that has an easy-to-use implementation
of BERT-embeddings for keyword/keyphrase extraction, let me know! I'll make sure to
add a reference to this repo.</p>
</article>
 </div>||||I|||| Skip to content
 Toggle navigation
 Sign up
 * Product
 + Actions
 Automate any workflow
 + Packages
 Host and manage packages
 + Security
 Find and fix vulnerabilities
 + Codespaces
 Instant dev environments
 + Copilot
 Write better code with AI
 + Code review
 Manage code changes
 + Issues
 Plan and track work
 + Discussions
 Collaborate outside of code
 + Explore
 + All features
 + Documentation
 + GitHub Skills
 + Blog
 * Solutions
 + For
 + Enterprise
 + Teams
 + Startups
 + Education
 + By Solution
 + CI/CD & Automation
 + DevOps
 + DevSecOps
 + Case Studies
 + Customer Stories
 + Resources
 * Open Source
 + GitHub Sponsors
 Fund open source developers
 + The ReadME Project
 GitHub community articles
 + Repositories
 + Topics
 + Trending
 + Collections
 * Pricing
 * In this repository All GitHub ↵
 Jump to ↵
 * No suggested jump to results
 * In this repository All GitHub ↵
 Jump to ↵
 * In this user All GitHub ↵
 Jump to ↵
 * In this repository All GitHub ↵
 Jump to ↵
 Sign in
 Sign up
 {{ message }}
 MaartenGr / KeyBERT Public
 * Notifications
 * Fork 258
 * Star 2.2k

 Minimal keyword extraction with BERT

 maartengr.github.io/keybert/

 License

 MIT license
 2.2k stars 258 forks
 Star
 Notifications
 * Code
 * Issues 20
 * Pull requests 1
 * Actions
 * Projects 0
 * Security
 * Insights
 More
 * Code
 * Issues
 * Pull requests
 * Actions
 * Projects
 * Security
 * Insights

 MaartenGr/KeyBERT

 This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
 master
 Switch branches/tags
 Branches Tags
 Could not load branches
 Nothing to show
 {{ refName }} default
 View all branches
 Could not load tags
 Nothing to show
 {{ refName }} default
 View all tags

 Name already in use

 A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
 Cancel Create
 10 branches 10 tags
 Code
 * Local
 * Codespaces
 * Clone
 HTTPS GitHub CLI

 Use Git or checkout with SVN using the web URL.

 Work fast with our official CLI. Learn more.

 * Open with GitHub Desktop
 * Download ZIP

 Sign In Required

 Please sign in to use Codespaces.

 Launching GitHub Desktop

 If nothing happens, download GitHub Desktop and try again.

 Launching GitHub Desktop

 If nothing happens, download GitHub Desktop and try again.

 Launching Xcode

 If nothing happens, download Xcode and try again.

 Launching Visual Studio Code

 Your codespace will open once ready.

 There was a problem preparing your codespace, please try again.

 Latest commit

 shengbo-ma Enable Guided KeyBERT for both local and global seed keywords (#152)
 …
 f91c502 Jan 18, 2023
 Enable Guided KeyBERT for both local and global seed keywords (#152)
 f91c502

 Git stats

 * 29 commits

 Files

 Permalink
 Failed to load latest commit information.
 Type
 Name
 Latest commit message
 Commit time
 .github/ workflows
 Enable Guided KeyBERT for both local and global seed keywords (#152)
 January 18, 2023 09:31
 docs
 v0.7 (#135)
 November 3, 2022 09:22
 images
 v0.4 (#43)
 June 30, 2021 11:47
 keybert
 Enable Guided KeyBERT for both local and global seed keywords (#152)
 January 18, 2023 09:31
 tests
 Enable Guided KeyBERT for both local and global seed keywords (#152)
 January 18, 2023 09:31
 theme
 Init commit
 October 22, 2020 14:21
 .flake8
 v0.5.1 (#97)
 March 31, 2022 09:20
 .gitattributes
 Init commit
 October 22, 2020 14:21
 .gitignore
 Enable Guided KeyBERT for both local and global seed keywords (#152)
 January 18, 2023 09:31
 .pre-commit-config.yaml
 v0.5.1 (#97)
 March 31, 2022 09:20
 LICENSE
 v0.5.1 (#97)
 March 31, 2022 09:20
 Makefile
 Add MMR (#4)
 October 27, 2020 10:24
 README.md
 v0.6.0 (#120)
 July 27, 2022 16:17
 mkdocs.yml
 v0.7 (#135)
 November 3, 2022 09:22
 setup.py
 v0.7 (#135)
 November 3, 2022 09:22
 View code
 KeyBERT Table of Contents 1. About the Project 2. Getting Started 2.1. Installation 2.2. Usage 2.3. Max Sum Distance 2.4. Maximal Marginal Relevance 2.5. Embedding Models Citation References

 README.md

 KeyBERT

 KeyBERT is a minimal and easy-to-use keyword extraction technique that leverages BERT embeddings to create keywords and keyphrases that are most similar to a document.

 Corresponding medium post can be found here.

 Table of Contents

 1. About the Project
 2. Getting Started
 2.1. Installation
 2.2. Basic Usage
 2.3. Max Sum Distance
 2.4. Maximal Marginal Relevance
 2.5. Embedding Models

 1. About the Project

 Back to ToC

 Although there are already many methods available for keyword generation (e.g., Rake, YAKE!, TF-IDF, etc.) I wanted to create a very basic, but powerful method for extracting keywords and keyphrases. This is where KeyBERT comes in! Which uses BERT-embeddings and simple cosine similarity to find the sub-phrases in a document that are the most similar to the document itself.

 First, document embeddings are extracted with BERT to get a document-level representation. Then, word embeddings are extracted for N-gram words/phrases. Finally, we use cosine similarity to find the words/phrases that are the most similar to the document. The most similar words could then be identified as the words that best describe the entire document.

 KeyBERT is by no means unique and is created as a quick and easy method for creating keywords and keyphrases. Although there are many great papers and solutions out there that use BERT-embeddings (e.g., 1, 2, 3, ), I could not find a BERT-based solution that did not have to be trained from scratch and could be used for beginners (correct me if I'm wrong!). Thus, the goal was a pip install keybert and at most 3 lines of code in usage.

 2. Getting Started

 Back to ToC

 2.1. Installation

 Installation can be done using pypi:

 pip install keybert
 

 You may want to install more depending on the transformers and language backends that you will be using. The possible installations are:

 pip install keybert[flair]
 pip install keybert[gensim]
 pip install keybert[spacy]
 pip install keybert[use]
 

 2.2. Usage

 The most minimal example can be seen below for the extraction of keywords:

 from keybert import KeyBERT
 
 doc = """
 Supervised learning is the machine learning task of learning a function that
 maps an input to an output based on example input-output pairs. It infers a
 function from labeled training data consisting of a set of training examples.
 In supervised learning, each example is a pair consisting of an input object
 (typically a vector) and a desired output value (also called the supervisory signal).
 A supervised learning algorithm analyzes the training data and produces an inferred function,
 which can be used for mapping new examples. An optimal scenario will allow for the
 algorithm to correctly determine the class labels for unseen instances. This requires
 the learning algorithm to generalize from the training data to unseen situations in a
 'reasonable' way (see inductive bias).
 """
 kw_model = KeyBERT()
 keywords = kw_model.extract_keywords(doc)

 You can set keyphrase_ngram_range to set the length of the resulting keywords/keyphrases:

 >>> kw_model.extract_keywords(doc, keyphrase_ngram_range=(1, 1), stop_words=None)
 [('learning', 0.4604),
 ('algorithm', 0.4556),
 ('training', 0.4487),
 ('class', 0.4086),
 ('mapping', 0.3700)]

 To extract keyphrases, simply set keyphrase_ngram_range to (1, 2) or higher depending on the number of words you would like in the resulting keyphrases:

 >>> kw_model.extract_keywords(doc, keyphrase_ngram_range=(1, 2), stop_words=None)
 [('learning algorithm', 0.6978),
 ('machine learning', 0.6305),
 ('supervised learning', 0.5985),
 ('algorithm analyzes', 0.5860),
 ('learning function', 0.5850)]

 We can highlight the keywords in the document by simply setting highlight:

 keywords = kw_model.extract_keywords(doc, highlight=True)

 NOTE: For a full overview of all possible transformer models see sentence-transformer. I would advise either "all-MiniLM-L6-v2" for English documents or "paraphrase-multilingual-MiniLM-L12-v2" for multi-lingual documents or any other language.

 2.3. Max Sum Distance

 To diversify the results, we take the 2 x top_n most similar words/phrases to the document. Then, we take all top_n combinations from the 2 x top_n words and extract the combination that are the least similar to each other by cosine similarity.

 >>> kw_model.extract_keywords(doc, keyphrase_ngram_range=(3, 3), stop_words='english',
 use_maxsum=True, nr_candidates=20, top_n=5)
 [('set training examples', 0.7504),
 ('generalize training data', 0.7727),
 ('requires learning algorithm', 0.5050),
 ('supervised learning algorithm', 0.3779),
 ('learning machine learning', 0.2891)]

 2.4. Maximal Marginal Relevance

 To diversify the results, we can use Maximal Margin Relevance (MMR) to create keywords / keyphrases which is also based on cosine similarity. The results with high diversity:

 >>> kw_model.extract_keywords(doc, keyphrase_ngram_range=(3, 3), stop_words='english',
 use_mmr=True, diversity=0.7)
 [('algorithm generalize training', 0.7727),
 ('labels unseen instances', 0.1649),
 ('new examples optimal', 0.4185),
 ('determine class labels', 0.4774),
 ('supervised learning algorithm', 0.7502)]

 The results with low diversity:

 >>> kw_model.extract_keywords(doc, keyphrase_ngram_range=(3, 3), stop_words='english',
 use_mmr=True, diversity=0.2)
 [('algorithm generalize training', 0.7727),
 ('supervised learning algorithm', 0.7502),
 ('learning machine learning', 0.7577),
 ('learning algorithm analyzes', 0.7587),
 ('learning algorithm generalize', 0.7514)]

 2.5. Embedding Models

 KeyBERT supports many embedding models that can be used to embed the documents and words:

 * Sentence-Transformers
 * Flair
 * Spacy
 * Gensim
 * USE

 Click here for a full overview of all supported embedding models.

 Sentence-Transformers
 You can select any model from sentence-transformers here and pass it through KeyBERT with model:

 from keybert import KeyBERT
 kw_model = KeyBERT(model='all-MiniLM-L6-v2')

 Or select a SentenceTransformer model with your own parameters:

 from keybert import KeyBERT
 from sentence_transformers import SentenceTransformer
 
 sentence_model = SentenceTransformer("all-MiniLM-L6-v2")
 kw_model = KeyBERT(model=sentence_model)

 Flair
 Flair allows you to choose almost any embedding model that is publicly available. Flair can be used as follows:

 from keybert import KeyBERT
 from flair.embeddings import TransformerDocumentEmbeddings
 
 roberta = TransformerDocumentEmbeddings('roberta-base')
 kw_model = KeyBERT(model=roberta)

 You can select any 🤗 transformers model here.

 Citation

 To cite KeyBERT in your work, please use the following bibtex reference:

 @misc{grootendorst2020keybert,
 author = {Maarten Grootendorst},
 title = {KeyBERT: Minimal keyword extraction with BERT.},
 year = 2020,
 publisher = {Zenodo},
 version = {v0.3.0},
 doi = {10.5281/zenodo.4461265},
 url = {https://doi.org/10.5281/zenodo.4461265}
 }

 References

 Below, you can find several resources that were used for the creation of KeyBERT but most importantly, these are amazing resources for creating impressive keyword extraction models:

 Papers:

 * Sharma, P., & Li, Y. (2019). Self-Supervised Contextual Keyword and Keyphrase Retrieval with Self-Labelling.

 Github Repos:

 * https://github.com/thunlp/BERT-KPE
 * https://github.com/ibatra/BERT-Keyword-Extractor
 * https://github.com/pranav-ust/BERT-keyphrase-extraction
 * https://github.com/swisscom/ai-research-keyphrase-extraction

 MMR: The selection of keywords/keyphrases was modeled after:

 * https://github.com/swisscom/ai-research-keyphrase-extraction

 NOTE: If you find a paper or github repo that has an easy-to-use implementation of BERT-embeddings for keyword/keyphrase extraction, let me know! I'll make sure to add a reference to this repo.

 About

 Minimal keyword extraction with BERT

 MaartenGr.github.io/KeyBERT/

 Topics

 mmr keyword-extraction bert keyphrase-extraction

 Resources

 Readme

 License

 MIT license

 Stars

 2.2k stars

 Watchers

 27 watching

 Forks

 258 forks

 Releases 10

 v0.7.0 Latest
 Nov 3, 2022
 + 9 releases

 Packages 0

 No packages published

 Contributors 8

 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 

 Languages

 * Python 99.4%
 * Makefile 0.6%

 Footer

 © 2023 GitHub, Inc.

 Footer navigation

 * Terms
 * Privacy
 * Security
 * Status
 * Docs
 * Contact GitHub
 * Pricing
 * API
 * Training
 * Blog
 * About
 You can’t perform that action at this time.
 You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
KeyBERT
KeyBERT is a minimal and easy-to-use keyword extraction technique that leverages BERT embeddings to
create keywords and keyphrases that are most similar to a document.
Corresponding medium post can be found here.
Table of Contents
 
 About the Project 
 Getting Started 
2.1. Installation 
2.2. Basic Usage 
2.3. Max Sum Distance 
2.4. Maximal Marginal Relevance 
2.5. Embedding Models 
 
1. About the Project
 Back to ToC 
Although there are already many methods available for keyword generation
(e.g.,
Rake,
YAKE!, TF-IDF, etc.)
I wanted to create a very basic, but powerful method for extracting keywords and keyphrases.
This is where KeyBERT comes in! Which uses BERT-embeddings and simple cosine similarity
to find the sub-phrases in a document that are the most similar to the document itself.
First, document embeddings are extracted with BERT to get a document-level representation.
Then, word embeddings are extracted for N-gram words/phrases. Finally, we use cosine similarity
to find the words/phrases that are the most similar to the document. The most similar words could
then be identified as the words that best describe the entire document.
KeyBERT is by no means unique and is created as a quick and easy method
for creating keywords and keyphrases. Although there are many great
papers and solutions out there that use BERT-embeddings
(e.g.,
1,
2,
3,
), I could not find a BERT-based solution that did not have to be trained from scratch and
could be used for beginners (correct me if I'm wrong!).
Thus, the goal was a pip install keybert and at most 3 lines of code in usage.
2. Getting Started
 Back to ToC 
2.1. Installation
Installation can be done using pypi:
You may want to install more depending on the transformers and language backends that you will be using. The possible installations are:
  pip install keybert[flair]
pip install keybert[gensim]
pip install keybert[spacy]
pip install keybert[use]
  
2.2. Usage
The most minimal example can be seen below for the extraction of keywords:
 from keybert import KeyBERT
doc = """
 Supervised learning is the machine learning task of learning a function that
 maps an input to an output based on example input-output pairs. It infers a
 function from labeled training data consisting of a set of training examples.
 In supervised learning, each example is a pair consisting of an input object
 (typically a vector) and a desired output value (also called the supervisory signal).
 A supervised learning algorithm analyzes the training data and produces an inferred function,
 which can be used for mapping new examples. An optimal scenario will allow for the
 algorithm to correctly determine the class labels for unseen instances. This requires
 the learning algorithm to generalize from the training data to unseen situations in a
 'reasonable' way (see inductive bias).
 """
kw_model = KeyBERT()
keywords = kw_model.extract_keywords(doc) 
You can set keyphrase_ngram_range to set the length of the resulting keywords/keyphrases:
 &gt;&gt;&gt; kw_model.extract_keywords(doc, keyphrase_ngram_range=(1, 1), stop_words=None)
[('learning', 0.4604),
 ('algorithm', 0.4556),
 ('training', 0.4487),
 ('class', 0.4086),
 ('mapping', 0.3700)] 
To extract keyphrases, simply set keyphrase_ngram_range to (1, 2) or higher depending on the number
of words you would like in the resulting keyphrases:
 &gt;&gt;&gt; kw_model.extract_keywords(doc, keyphrase_ngram_range=(1, 2), stop_words=None)
[('learning algorithm', 0.6978),
 ('machine learning', 0.6305),
 ('supervised learning', 0.5985),
 ('algorithm analyzes', 0.5860),
 ('learning function', 0.5850)] 
We can highlight the keywords in the document by simply setting highlight:
 keywords = kw_model.extract_keywords(doc, highlight=True) 
   
 NOTE: For a full overview of all possible transformer models see sentence-transformer.
I would advise either "all-MiniLM-L6-v2" for English documents or "paraphrase-multilingual-MiniLM-L12-v2"
for multi-lingual documents or any other language.
2.3. Max Sum Distance
To diversify the results, we take the 2 x top_n most similar words/phrases to the document.
Then, we take all top_n combinations from the 2 x top_n words and extract the combination
that are the least similar to each other by cosine similarity.
 &gt;&gt;&gt; kw_model.extract_keywords(doc, keyphrase_ngram_range=(3, 3), stop_words='english',
 use_maxsum=True, nr_candidates=20, top_n=5)
[('set training examples', 0.7504),
 ('generalize training data', 0.7727),
 ('requires learning algorithm', 0.5050),
 ('supervised learning algorithm', 0.3779),
 ('learning machine learning', 0.2891)] 
2.4. Maximal Marginal Relevance
To diversify the results, we can use Maximal Margin Relevance (MMR) to create
keywords / keyphrases which is also based on cosine similarity. The results
with high diversity:
 &gt;&gt;&gt; kw_model.extract_keywords(doc, keyphrase_ngram_range=(3, 3), stop_words='english',
 use_mmr=True, diversity=0.7)
[('algorithm generalize training', 0.7727),
 ('labels unseen instances', 0.1649),
 ('new examples optimal', 0.4185),
 ('determine class labels', 0.4774),
 ('supervised learning algorithm', 0.7502)] 
The results with low diversity:
 &gt;&gt;&gt; kw_model.extract_keywords(doc, keyphrase_ngram_range=(3, 3), stop_words='english',
 use_mmr=True, diversity=0.2)
[('algorithm generalize training', 0.7727),
 ('supervised learning algorithm', 0.7502),
 ('learning machine learning', 0.7577),
 ('learning algorithm analyzes', 0.7587),
 ('learning algorithm generalize', 0.7514)] 
2.5. Embedding Models
KeyBERT supports many embedding models that can be used to embed the documents and words:
Sentence-Transformers
Flair
Spacy
Gensim
USE
Click here for a full overview of all supported embedding models.
 Sentence-Transformers 
You can select any model from sentence-transformers here
and pass it through KeyBERT with model:
 from keybert import KeyBERT
kw_model = KeyBERT(model='all-MiniLM-L6-v2') 
Or select a SentenceTransformer model with your own parameters:
 from keybert import KeyBERT
from sentence_transformers import SentenceTransformer
sentence_model = SentenceTransformer("all-MiniLM-L6-v2")
kw_model = KeyBERT(model=sentence_model) 
 Flair 
Flair allows you to choose almost any embedding model that
is publicly available. Flair can be used as follows:
 from keybert import KeyBERT
from flair.embeddings import TransformerDocumentEmbeddings
roberta = TransformerDocumentEmbeddings('roberta-base')
kw_model = KeyBERT(model=roberta) 
You can select any 🤗 transformers model here.
Citation
To cite KeyBERT in your work, please use the following bibtex reference:
 @misc{grootendorst2020keybert,
 author = {Maarten Grootendorst},
 title = {KeyBERT: Minimal keyword extraction with BERT.},
 year = 2020,
 publisher = {Zenodo},
 version = {v0.3.0},
 doi = {10.5281/zenodo.4461265},
 url = {https://doi.org/10.5281/zenodo.4461265}
} 
References
Below, you can find several resources that were used for the creation of KeyBERT
but most importantly, these are amazing resources for creating impressive keyword extraction models:
 Papers:
Sharma, P., &amp; Li, Y. (2019). Self-Supervised Contextual Keyword and Keyphrase Retrieval with Self-Labelling. 
 Github Repos:
 https://github.com/thunlp/BERT-KPE 
 https://github.com/ibatra/BERT-Keyword-Extractor 
 https://github.com/pranav-ust/BERT-keyphrase-extraction 
 https://github.com/swisscom/ai-research-keyphrase-extraction 
 MMR:
The selection of keywords/keyphrases was modeled after:
 https://github.com/swisscom/ai-research-keyphrase-extraction 
 NOTE: If you find a paper or github repo that has an easy-to-use implementation
of BERT-embeddings for keyword/keyphrase extraction, let me know! I'll make sure to
add a reference to this repo.
GitHub - MaartenGr/KeyBERT: Minimal keyword extraction with BERT
https://github.com/MaartenGr/KeyBERT#installation
https://kavita-ganesan.com/python-keyword-extraction/
Python Keyword Extraction Tutorial using TF-IDF
2018-08-02
Kavita Ganesan
<div><div><main><article><div><p>In this era of use Deep Learning for everything, one may be wondering why you would even use TF-IDF for any task at all ?!! The truth is TF-IDF is easy to understand, easy to compute and is one of the most versatile statistic that shows the relative importance of a word or phrase in a document or a set of documents in comparison to the rest of your corpus.</p><p>Keywords are descriptive words or phrases that characterize your documents. For example, keywords from this article would be <code>tf-idf</code>,   <code>scikit-learn</code>, <code>keyword extraction</code>, <code>extract</code> and so on. These keywords are also referred to as topics in some applications.</p><p>TF-IDF can be used for a wide range of tasks including <a href="https://www.quora.com/Can-we-use-TF-IDF-for-text-classification-tasks"><em><strong>text classification</strong></em></a>, <a href="http://jonathanzong.com/blog/2013/02/02/k-means-clustering-with-tfidf-weights"><em><strong>clustering</strong></em></a> /<em><strong> topic-modeling</strong></em>, <em><strong>search</strong></em>, <em><strong>keyword extraction</strong></em> and a whole lot more.</p><p>In this article, you will learn how to perform keyword extraction using python, specifically using <strong>TF-IDF from the scikit-learn package</strong> to extract keywords from documents.</p><h2>Let’s get started with python keyword extraction</h2><p>I’m assuming that folks following this tutorial are already familiar with the concept of TF-IDF. If you are not, please familiarize yourself with the concept before reading on. There are a couple of <a href="https://www.youtube.com/results?search_query=tf-idf.">videos online</a> that give an intuitive explanation of what it is. For a more academic explanation I would recommend my <a href="https://www.coursera.org/lecture/text-retrieval/lesson-2-2-tf-transformation-W0NZe">Ph.D advisor’s explanation</a>.</p><h2>Dataset</h2><p>In this keyword extraction tutorial, we’ll be using a stack overflow dataset which is a bit noisy and simulates what you could be dealing with in real life.</p><p>We will be using two files, one file, <code>stackoverflow-data-idf.json</code> has 20,000 posts and is used to compute the Inverse Document Frequency (IDF) and another file, <code>stackoverflow-test.json</code> has 500 posts and we would use that as a test set for us to extract keywords from. This dataset is based on the publicly available <a href="https://cloud.google.com/bigquery/public-data/stackoverflow">stack overflow dump from Google’s Big Query</a>.</p><p>The first thing we’ll do is to take a peek at our dataset. The code below reads a one per line json string from <code>data/stackoverflow-data-idf.json</code> into a pandas data frame and prints out its schema and total number of posts. Here, <code>lines=True</code> simply means we are treating each line in the text file as a separate json string.</p><pre><code>import pandas as pd
# read json into a dataframe
df_idf=pd.read_json("data/stackoverflow-data-idf.json",lines=True)
# print schema
print("Schema:\n\n",df_idf.dtypes)
print("Number of questions,columns=",df_idf.shape)</code></pre><pre>Schema:
accepted_answer_id float64
answer_count int64
body object
comment_count int64
community_owned_date object
creation_date object
favorite_count float64
id int64
last_activity_date object
last_edit_date object
last_editor_display_name object
last_editor_user_id float64
owner_display_name object
owner_user_id float64
post_type_id int64
score int64
tags object
title object
view_count int64
dtype: object
Number of questions,columns= (20000, 19)
</pre><p>Notice that this stack overflow dataset contains 19 fields including post title, body, tags, dates and other metadata which we don’t quite need for this tutorial. What we are mostly interested in for this tutorial, is the body and title which will become our source of text for keyword extraction. We will now create a field that combines both <code>body</code> and <code>title</code> so we have it in one field. We will also print the second text entry in our new field just to see what the text looks like.</p><pre><code>
import re
def pre_process(text):
 
 # lowercase
 text=text.lower()
 
 #remove tags
 text=re.sub("","",text)
 
 # remove special characters and digits
 text=re.sub("(\\d|\\W)+"," ",text)
 
 return text
df_idf['text'] = df_idf['title'] + df_idf['body']
df_idf['text'] = df_idf['text'].apply(lambda x:pre_process(x))
#show the second 'text' just for fun
df_idf['text'][2]
</code></pre><p><br /> The text above is essentially a combination of the title and body of a stack overflow post. Hmmm, this doesn’t look very readable, does it? Well, that’s because we are cleaning the text after we concatenated the two fields (line 18). All of the cleaning happens in <code>pre_process(..)</code>. You can do a lot more stuff in <code>pre_process(..)</code>, such as eliminate all code sections, normalize the words to its root, etc, but for simplicity we perform only some mild pre-processing.</p><h2>Creating Vocabulary and Word Counts for IDF</h2><p>We now need to create the vocabulary and start the counting process. We can use the CountVectorizer to create a vocabulary from all the text in our <code>df_idf['text']</code> followed by the counts of words in the vocabulary (see: <a href="https://kavita-ganesan.com/how-to-use-countvectorizer/">usage examples for CountVectorizer</a>).</p><pre><code>
from sklearn.feature_extraction.text import CountVectorizer
import re
def get_stop_words(stop_file_path):
 """load stop words """
 
 with open(stop_file_path, 'r', encoding="utf-8") as f:
 stopwords = f.readlines()
 stop_set = set(m.strip() for m in stopwords)
 return frozenset(stop_set)
#load a set of stop words
stopwords=get_stop_words("resources/stopwords.txt")
#get the text column 
docs=df_idf['text'].tolist()
#create a vocabulary of words, 
#ignore words that appear in 85% of documents, 
#eliminate stop words
cv=CountVectorizer(max_df=0.85,stop_words=stopwords)
word_count_vector=cv.fit_transform(docs)
</code></pre><p>While <code>cv.fit(...)</code> would only create the vocabulary, <code>cv.fit_transform(...)</code> creates the vocabulary and returns a term-document matrix which is what we want. With this, each column in the matrix represents a word in the vocabulary while each row represents the document in our dataset where the values in this case are the word counts. Note that with this representation, counts of some words could be 0 if the word did not appear in the corresponding document.</p><p>Notice that in the code above, we are passing two parameters to <a href="https://kavita-ganesan.com/how-to-use-countvectorizer/">CountVectorizer</a>, <code>max_df</code> and <code>stop_words</code>. The first is just to say ignore all words that have appeared in 85% of the documents, since those may be unimportant. The later, is a custom stop words list. You can also use stop words that are native to sklearn by setting <code>stop_words='english'</code>, but I personally find this to be quite limited. I used a custom <a href="http://kavita-ganesan.com/what-are-stop-words/">stop word</a> list used for this tutorial.</p><p>The resulting shape of <code>word_count_vector</code> is (20000,124901) since we have 20,000 documents in our dataset (the rows) and the vocabulary size is 124,901. In some text mining applications such as clustering and text classification we typically limit the size of the vocabulary. It’s really easy to do this by setting <code>max_features=vocab_size</code> when instantiating CountVectorizer. For this tutorial let’s limit our vocabulary size to 10,000.</p><pre><code>
cv=CountVectorizer(max_df=0.85,stop_words=stopwords,max_features=10000)
word_count_vector=cv.fit_transform(docs)
</code></pre><p>Now, let’s look at 10 words from our vocabulary.</p><pre><code>
list(cv.vocabulary_.keys())[:10]
</code></pre><pre>['serializing',
 'private',
 'struct',
 'public',
 'class',
 'contains',
 'properties',
 'string',
 'serialize',
 'attempt']
</pre><p>Sweet, these are mostly programming related.</p><h2>TfidfTransformer to Compute Inverse Document Frequency (IDF)</h2><p>It’s now time to compute the IDF values. In the code below, we are essentially taking the sparse matrix from CountVectorizer (<code>word_count_vector</code>) to generate the IDF when you invoke <code>tfidf_transformer.fit(...)</code>(see: <a href="http://kavita-ganesan.com/tfidftransformer-tfidfvectorizer-usage-differences/#.XM9aY9NKiql">basic usage example of tfidftransformer and tfidfvectorizer</a>)</p><pre><code>
from sklearn.feature_extraction.text import TfidfTransformer
tfidf_transformer=TfidfTransformer(smooth_idf=True,use_idf=True)
tfidf_transformer.fit(word_count_vector)
</code></pre><p>An extremely important point to note here is that the <a href="https://kavita-ganesan.com/what-is-inverse-document-frequency/#.XcBgIpNKjm0">IDF</a> should always be based on a large corpora and should be representative of texts you would be using to extract keywords. This is why we are using texts from <code>20,000</code> stack overflow posts to compute the IDF instead of just a handful. I’ve seen several articles on the Web that compute the IDF using a handful of documents. You will defeat the whole purpose of IDF weighting if its not based on a large corpora as (a) your vocabulary becomes too small and (b) you have limited ability to observe the behavior of words that you do know about.</p><h2>Computing TF-IDF and Extracting Keywords</h2><p>Once we have our IDF computed, we are now ready to compute TF-IDF and then extract top keywords from the TF-IDF vectors. In this example, we will extract top keywords for the questions in <code>data/stackoverflow-test.json</code>. This data file has 500 questions with fields identical to that of <code>data/stackoverflow-data-idf.json</code> as we saw above. We will start by reading our test file, extracting the necessary fields (title and body) and getting the texts into a list.</p><pre><code>
# read test docs into a dataframe and concatenate title and body
df_test=pd.read_json("data/stackoverflow-test.json",lines=True)
df_test['text'] = df_test['title'] + df_test['body']
df_test['text'] =df_test['text'].apply(lambda x:pre_process(x))
# get test docs into a list
docs_test=df_test['text'].tolist()
</code></pre><p>The next step is to compute the tf-idf value for a given document in our test set by invoking <code>tfidf_transformer.transform(...)</code>. This generates a vector of tf-idf scores. Next, we sort the words in the vector in descending order of tf-idf values and then iterate over to extract the top-n keywords. In the example below, we are extracting keywords for the first document in our test set.</p><pre><code>
# you only needs to do this once, this is a mapping of index to 
feature_names=cv.get_feature_names()
# get the document that we want to extract keywords from
doc=docs_test[0]
#generate tf-idf for the given document
tf_idf_vector=tfidf_transformer.transform(cv.transform([doc]))
#sort the tf-idf vectors by descending order of scores
sorted_items=sort_coo(tf_idf_vector.tocoo())
#extract only the top n; n here is 10
keywords=extract_topn_from_vector(feature_names,sorted_items,10)
# now print the results
print("\n=====Doc=====")
print(doc)
print("\n===Keywords===")
for k in keywords:
 print(k,keywords[k])
</code></pre><pre><code>
def sort_coo(coo_matrix):
 tuples = zip(coo_matrix.col, coo_matrix.data)
 return sorted(tuples, key=lambda x: (x[1], x[0]), reverse=True)
def extract_topn_from_vector(feature_names, sorted_items, topn=10):
 """get the feature names and tf-idf score of top n items"""
 
 #use only topn items from vector
 sorted_items = sorted_items[:topn]
 score_vals = []
 feature_vals = []
 
 # word index and corresponding tf-idf score
 for idx, score in sorted_items:
 
 #keep track of feature name and its corresponding score
 score_vals.append(round(score, 3))
 feature_vals.append(feature_names[idx])
 #create a tuples of feature,score
 #results = zip(feature_vals,score_vals)
 results= {}
 for idx in range(len(feature_vals)):
 results[feature_vals[idx]]=score_vals[idx]
 
 return results
</code></pre><p>The <code>sort_coo(...)</code> method essentially sorts the values in the vector while preserving the column index. Once you have the column index then its really easy to look-up the corresponding word value as you would see in <code>extract_topn_from_vector(...)</code> where we do <code>feature_vals.append(feature_names[idx])</code>.</p><h2>Example Results</h2><p>In this section, you will see some of the stack overflow questions followed by the top-10 keywords generated using the code above. Note that these questions are from the <code>stackoverflow-test.json</code> data file.</p><h4>Question about Eclipse Plugin integration</h4><p><br /> From the keywords above, the top keywords actually make sense, it talks about <code>eclipse</code>, <code>maven</code>, <code>integrate</code>, <code>war</code> and <code>tomcat</code> which are all unique to this specific question. There are a couple of keywords that could have been eliminated such as <code>possibility</code> and perhaps even <code>project</code> and you can further fine-tune what shows up on top by adding more common words to your stop list and you can even create your own set of stop list, <a href="http://kavita-ganesan.com/tips-for-constructing-custom-stop-word-lists/">very specific to your domain</a>.</p><p>Now let’s look at another example.</p><h4>Question about SQL Import</h4><p></p><p>Even with all the html tags, because of the pre-processing, we are able to extract some pretty nice keywords here. The last word <code>appropriately</code> would qualify as a stop word. You can keep running different examples to get ideas of how to fine-tune the results.</p><p>Wow! Now you can extract important keywords from any type of text!</p><h2>Some tips and tricks</h2><ol><li>You can easily <a href="https://stackoverflow.com/questions/29788047/keep-tfidf-result-for-predicting-new-content-using-scikit-for-python">save the resulting CountVectorizer and TfidfTransformer</a> and load them back for use at a later time.</li><li>Instead of using CountVectorizer followed by TfidfTransformer, you can directly use TfidfVectorizer by itself. This is equivalent to CountVectorizer followed by TfidfTransformer.</li><li>In this example, we computed the tf-idf matrix for each document of interest and then extracted top terms from it. What you could also do is first apply<code>tfidf_transformer.transform(docs_test)</code> which will generate a tf-idf matrix for all documents in <code>docs_test</code> at one go and then iterate over the resulting vectors to extract top keywords. The first approach is useful if you have one document coming in at a time. The second approach is more suitable when you want keywords from a fairly large set of documents.</li></ol><h3>See Also: <a href="https://kavita-ganesan.com/tfidftransformer-tfidfvectorizer-usage-differences/">How to correctly use TFIDFTransformer and TFIDFVectorizer?</a></h3><h2>Resources for python keyword extraction</h2><ul><li>Get the <a href="https://github.com/kavgan/nlp-in-practice/tree/master/tf-idf">code samples</a> for this tutorial</li><li><a href="http://kavita-ganesan.com/how-to-use-tfidftransformer-tfidfvectorizer-and-whats-the-difference/">Explanation on using TfIdftransformer and Tfidfvectorizer</a></li><li>Stack overflow data on <a href="https://cloud.google.com/bigquery/public-data/stackoverflow">Google’s BigQuery</a></li></ul> </div></article></main></div>||||I||||Skip to content
 Main Menu
 * HOME
 * ABOUT Menu Toggle
 + About Kavita
 + Patents
 + Academic Publications
 + Open Source
 * RESOURCES Menu Toggle
 + AI Technical Resources Menu Toggle
 o Data Science Python Cheat Sheet
 o NLP/ML Tutorials
 o NLP Starter Code
 * SPEAKING
 * PRESS
 * AI BOOK
 * BLOG
 Subscribe

 Python Keyword Extraction Tutorial using TF-IDF

 By Kavita Ganesan / AI Implementation , Hands-On NLP , Text Mining

 In this era of use Deep Learning for everything, one may be wondering why you would even use TF-IDF for any task at all ?!! The truth is TF-IDF is easy to understand, easy to compute and is one of the most versatile statistic that shows the relative importance of a word or phrase in a document or a set of documents in comparison to the rest of your corpus.

 Keywords are descriptive words or phrases that characterize your documents. For example, keywords from this article would be tf-idf, scikit-learn, keyword extraction, extract and so on. These keywords are also referred to as topics in some applications.

 TF-IDF can be used for a wide range of tasks including text classification, clustering / topic-modeling, search, keyword extraction and a whole lot more.

 In this article, you will learn how to perform keyword extraction using python, specifically using TF-IDF from the scikit-learn package to extract keywords from documents.

 Let’s get started with python keyword extraction

 I’m assuming that folks following this tutorial are already familiar with the concept of TF-IDF. If you are not, please familiarize yourself with the concept before reading on. There are a couple of videos online that give an intuitive explanation of what it is. For a more academic explanation I would recommend my Ph.D advisor’s explanation.

 Dataset

 In this keyword extraction tutorial, we’ll be using a stack overflow dataset which is a bit noisy and simulates what you could be dealing with in real life.

 We will be using two files, one file, stackoverflow-data-idf.json has 20,000 posts and is used to compute the Inverse Document Frequency (IDF) and another file, stackoverflow-test.json has 500 posts and we would use that as a test set for us to extract keywords from. This dataset is based on the publicly available stack overflow dump from Google’s Big Query.

 The first thing we’ll do is to take a peek at our dataset. The code below reads a one per line json string from data/stackoverflow-data-idf.json into a pandas data frame and prints out its schema and total number of posts. Here, lines=True simply means we are treating each line in the text file as a separate json string.

 import pandas as pd
 # read json into a dataframe
 df_idf=pd.read_json("data/stackoverflow-data-idf.json",lines=True)
 # print schema
 print("Schema:\n\n",df_idf.dtypes)
 print("Number of questions,columns=",df_idf.shape)
 Schema:
 accepted_answer_id float64
 answer_count int64
 body object
 comment_count int64
 community_owned_date object
 creation_date object
 favorite_count float64
 id int64
 last_activity_date object
 last_edit_date object
 last_editor_display_name object
 last_editor_user_id float64
 owner_display_name object
 owner_user_id float64
 post_type_id int64
 score int64
 tags object
 title object
 view_count int64
 dtype: object
 Number of questions,columns= (20000, 19)
 

 Notice that this stack overflow dataset contains 19 fields including post title, body, tags, dates and other metadata which we don’t quite need for this tutorial. What we are mostly interested in for this tutorial, is the body and title which will become our source of text for keyword extraction. We will now create a field that combines both body and title so we have it in one field. We will also print the second text entry in our new field just to see what the text looks like.

 
 import re
 def pre_process(text):
 
 # lowercase
 text=text.lower()
 
 #remove tags
 text=re.sub("","",text)
 
 # remove special characters and digits
 text=re.sub("(\\d|\\W)+"," ",text)
 
 return text
 df_idf['text'] = df_idf['title'] + df_idf['body']
 df_idf['text'] = df_idf['text'].apply(lambda x:pre_process(x))
 #show the second 'text' just for fun
 df_idf['text'][2]
 


 The text above is essentially a combination of the title and body of a stack overflow post. Hmmm, this doesn’t look very readable, does it? Well, that’s because we are cleaning the text after we concatenated the two fields (line 18). All of the cleaning happens in pre_process(..). You can do a lot more stuff in pre_process(..), such as eliminate all code sections, normalize the words to its root, etc, but for simplicity we perform only some mild pre-processing.

 Creating Vocabulary and Word Counts for IDF

 We now need to create the vocabulary and start the counting process. We can use the CountVectorizer to create a vocabulary from all the text in our df_idf['text'] followed by the counts of words in the vocabulary (see: usage examples for CountVectorizer).

 
 from sklearn.feature_extraction.text import CountVectorizer
 import re
 def get_stop_words(stop_file_path):
 """load stop words """
 
 with open(stop_file_path, 'r', encoding="utf-8") as f:
 stopwords = f.readlines()
 stop_set = set(m.strip() for m in stopwords)
 return frozenset(stop_set)
 #load a set of stop words
 stopwords=get_stop_words("resources/stopwords.txt")
 #get the text column 
 docs=df_idf['text'].tolist()
 #create a vocabulary of words, 
 #ignore words that appear in 85% of documents, 
 #eliminate stop words
 cv=CountVectorizer(max_df=0.85,stop_words=stopwords)
 word_count_vector=cv.fit_transform(docs)
 

 While cv.fit(...) would only create the vocabulary, cv.fit_transform(...) creates the vocabulary and returns a term-document matrix which is what we want. With this, each column in the matrix represents a word in the vocabulary while each row represents the document in our dataset where the values in this case are the word counts. Note that with this representation, counts of some words could be 0 if the word did not appear in the corresponding document.

 Notice that in the code above, we are passing two parameters to CountVectorizer, max_df and stop_words. The first is just to say ignore all words that have appeared in 85% of the documents, since those may be unimportant. The later, is a custom stop words list. You can also use stop words that are native to sklearn by setting stop_words='english', but I personally find this to be quite limited. I used a custom stop word list used for this tutorial.

 The resulting shape of word_count_vector is (20000,124901) since we have 20,000 documents in our dataset (the rows) and the vocabulary size is 124,901. In some text mining applications such as clustering and text classification we typically limit the size of the vocabulary. It’s really easy to do this by setting max_features=vocab_size when instantiating CountVectorizer. For this tutorial let’s limit our vocabulary size to 10,000.

 
 cv=CountVectorizer(max_df=0.85,stop_words=stopwords,max_features=10000)
 word_count_vector=cv.fit_transform(docs)
 

 Now, let’s look at 10 words from our vocabulary.

 
 list(cv.vocabulary_.keys())[:10]
 
 ['serializing',
 'private',
 'struct',
 'public',
 'class',
 'contains',
 'properties',
 'string',
 'serialize',
 'attempt']
 

 Sweet, these are mostly programming related.

 TfidfTransformer to Compute Inverse Document Frequency (IDF)

 It’s now time to compute the IDF values. In the code below, we are essentially taking the sparse matrix from CountVectorizer (word_count_vector) to generate the IDF when you invoke tfidf_transformer.fit(...)(see: basic usage example of tfidftransformer and tfidfvectorizer)

 
 from sklearn.feature_extraction.text import TfidfTransformer
 tfidf_transformer=TfidfTransformer(smooth_idf=True,use_idf=True)
 tfidf_transformer.fit(word_count_vector)
 

 An extremely important point to note here is that the IDF should always be based on a large corpora and should be representative of texts you would be using to extract keywords. This is why we are using texts from 20,000 stack overflow posts to compute the IDF instead of just a handful. I’ve seen several articles on the Web that compute the IDF using a handful of documents. You will defeat the whole purpose of IDF weighting if its not based on a large corpora as (a) your vocabulary becomes too small and (b) you have limited ability to observe the behavior of words that you do know about.

 Computing TF-IDF and Extracting Keywords

 Once we have our IDF computed, we are now ready to compute TF-IDF and then extract top keywords from the TF-IDF vectors. In this example, we will extract top keywords for the questions in data/stackoverflow-test.json. This data file has 500 questions with fields identical to that of data/stackoverflow-data-idf.json as we saw above. We will start by reading our test file, extracting the necessary fields (title and body) and getting the texts into a list.

 
 # read test docs into a dataframe and concatenate title and body
 df_test=pd.read_json("data/stackoverflow-test.json",lines=True)
 df_test['text'] = df_test['title'] + df_test['body']
 df_test['text'] =df_test['text'].apply(lambda x:pre_process(x))
 # get test docs into a list
 docs_test=df_test['text'].tolist()
 

 The next step is to compute the tf-idf value for a given document in our test set by invoking tfidf_transformer.transform(...). This generates a vector of tf-idf scores. Next, we sort the words in the vector in descending order of tf-idf values and then iterate over to extract the top-n keywords. In the example below, we are extracting keywords for the first document in our test set.

 
 # you only needs to do this once, this is a mapping of index to 
 feature_names=cv.get_feature_names()
 # get the document that we want to extract keywords from
 doc=docs_test[0]
 #generate tf-idf for the given document
 tf_idf_vector=tfidf_transformer.transform(cv.transform([doc]))
 #sort the tf-idf vectors by descending order of scores
 sorted_items=sort_coo(tf_idf_vector.tocoo())
 #extract only the top n; n here is 10
 keywords=extract_topn_from_vector(feature_names,sorted_items,10)
 # now print the results
 print("\n=====Doc=====")
 print(doc)
 print("\n===Keywords===")
 for k in keywords:
 print(k,keywords[k])
 
 
 def sort_coo(coo_matrix):
 tuples = zip(coo_matrix.col, coo_matrix.data)
 return sorted(tuples, key=lambda x: (x[1], x[0]), reverse=True)
 def extract_topn_from_vector(feature_names, sorted_items, topn=10):
 """get the feature names and tf-idf score of top n items"""
 
 #use only topn items from vector
 sorted_items = sorted_items[:topn]
 score_vals = []
 feature_vals = []
 
 # word index and corresponding tf-idf score
 for idx, score in sorted_items:
 
 #keep track of feature name and its corresponding score
 score_vals.append(round(score, 3))
 feature_vals.append(feature_names[idx])
 #create a tuples of feature,score
 #results = zip(feature_vals,score_vals)
 results= {}
 for idx in range(len(feature_vals)):
 results[feature_vals[idx]]=score_vals[idx]
 
 return results
 

 The sort_coo(...) method essentially sorts the values in the vector while preserving the column index. Once you have the column index then its really easy to look-up the corresponding word value as you would see in extract_topn_from_vector(...) where we do feature_vals.append(feature_names[idx]).

 Example Results

 In this section, you will see some of the stack overflow questions followed by the top-10 keywords generated using the code above. Note that these questions are from the stackoverflow-test.json data file.

 Question about Eclipse Plugin integration


 From the keywords above, the top keywords actually make sense, it talks about eclipse, maven, integrate, war and tomcat which are all unique to this specific question. There are a couple of keywords that could have been eliminated such as possibility and perhaps even project and you can further fine-tune what shows up on top by adding more common words to your stop list and you can even create your own set of stop list, very specific to your domain.

 Now let’s look at another example.

 Question about SQL Import

 Even with all the html tags, because of the pre-processing, we are able to extract some pretty nice keywords here. The last word appropriately would qualify as a stop word. You can keep running different examples to get ideas of how to fine-tune the results.

 Wow! Now you can extract important keywords from any type of text!

 Some tips and tricks

 1. You can easily save the resulting CountVectorizer and TfidfTransformer and load them back for use at a later time.
 2. Instead of using CountVectorizer followed by TfidfTransformer, you can directly use TfidfVectorizer by itself. This is equivalent to CountVectorizer followed by TfidfTransformer.
 3. In this example, we computed the tf-idf matrix for each document of interest and then extracted top terms from it. What you could also do is first applytfidf_transformer.transform(docs_test) which will generate a tf-idf matrix for all documents in docs_test at one go and then iterate over the resulting vectors to extract top keywords. The first approach is useful if you have one document coming in at a time. The second approach is more suitable when you want keywords from a fairly large set of documents.

 See Also: How to correctly use TFIDFTransformer and TFIDFVectorizer?

 Resources for python keyword extraction

 * Get the code samples for this tutorial
 * Explanation on using TfIdftransformer and Tfidfvectorizer
 * Stack overflow data on Google’s BigQuery

 Related

 Post navigation
 ← Previous Post
 Next Post →

 25 thoughts on “Python Keyword Extraction Tutorial using TF-IDF”

 Newer Comments

 Older Comments
 1. Mukunthan Lakshmanan
 September 4, 2020 at 11:14 am

 Hi, Thanks for your post. Can you please also explain or suggest means to find key phrases and not only key words. Key phrases, for example like, “business goals”, “key performance index”, “value addition”, etc. from a text object?

 Also please let me know, if you are available for any free-lance projects / training initiatives. Thanks.

 Reply
 2. Azi
 May 11, 2020 at 7:12 pm

 Thanks it is really helpfull

 Reply
 3. sapna sinha
 March 20, 2020 at 2:20 pm

 Hi Kavita, Thank you so much for this insightful article. I plan to cluster some unlabelled documents together based on their text content. Do you think if I use K Means for clustering tfidf vectorized text, I could get common keyterms for documents clustered together?

 Reply
 1. Kavita Ganesan
 March 20, 2020 at 3:51 pm

 Hi Sapna,

 Yes, but the quality of the clusters would depend on a lot on other factors. For example, the quality of the tf-idf vectors space itself, the number of clusters, iterations, seed clusters etc.

 Reply

 Newer Comments

 Older Comments

 Have a thought?

 Launch AI Initiatives that get results using Kavita's book, The Business Case for AI as a blueprint.

 DOWNLOAD PREVIEW

 "The author does a fantastic job breaking down some pretty complex concepts and uses relatable examples to keep you following along. I'm not a huge non-fiction person, let alone business books, but I could not put this one down. It's equal parts educational, fascinating, and weirdly thrilling."
 —Amazon Reader

 "This is one of the best books for demystifying AI from a business perspective without going too much into technical details. Typically there are too high-level books stating AI is the new electricity or books that go to discussions such as is Random Forest better than XGBoost. This is a perfect match in between!"
 —CEO @ DataDesign

 Latest Articles

 Before AI, Invest in A Big Data Strategy

 How To Generate Quality Training Data For Your Machine Learning Projects

 How Sentiment Analysis Keeps Your Brand in Check (and How to Get Started)

 How to Prevent Machine Learning Models from Failing in Practice?

 Follow Me On LinkedIn

 Connect on Social

 * 
 * 

 Search

 Search for:

 Contact

 To get in touch with Kavita, use her contact form or email kavita@opinosis.ai.

 Blog Posts

 * GPT-3: What is GPT-3 and what can it do for your business?
 * What went wrong with Tay, the Twitter bot that turned racist?
 * 3 Strategic Mistakes Leaders Can Easily Avoid When Thinking About AI Integration

 AI Strategy Book

 Get Kavita's latest AI book for business leaders.

 LEARN MORE
 Copyright © 2023 Kavita Ganesan, PhD
 * Home
 * About
 * Contact
In this era of use Deep Learning for everything, one may be wondering why you would even use TF-IDF for any task at all ?!! The truth is TF-IDF is easy to understand, easy to compute and is one of the most versatile statistic that shows the relative importance of a word or phrase in a document or a set of documents in comparison to the rest of your corpus. Keywords are descriptive words or phrases that characterize your documents. For example, keywords from this article would be tf-idf,   scikit-learn, keyword extraction, extract and so on. These keywords are also referred to as topics in some applications. TF-IDF can be used for a wide range of tasks including   text classification  ,   clustering   /  topic-modeling ,  search ,  keyword extraction  and a whole lot more. In this article, you will learn how to perform keyword extraction using python, specifically using TF-IDF from the scikit-learn package to extract keywords from documents. Let’s get started with python keyword extraction I’m assuming that folks following this tutorial are already familiar with the concept of TF-IDF. If you are not, please familiarize yourself with the concept before reading on. There are a couple of videos online that give an intuitive explanation of what it is. For a more academic explanation I would recommend my Ph.D advisor’s explanation. Dataset In this keyword extraction tutorial, we’ll be using a stack overflow dataset which is a bit noisy and simulates what you could be dealing with in real life. We will be using two files, one file, stackoverflow-data-idf.json has 20,000 posts and is used to compute the Inverse Document Frequency (IDF) and another file, stackoverflow-test.json has 500 posts and we would use that as a test set for us to extract keywords from. This dataset is based on the publicly available stack overflow dump from Google’s Big Query. The first thing we’ll do is to take a peek at our dataset. The code below reads a one per line json string from data/stackoverflow-data-idf.json into a pandas data frame and prints out its schema and total number of posts. Here, lines=True simply means we are treating each line in the text file as a separate json string.  import pandas as pd
# read json into a dataframe
df_idf=pd.read_json("data/stackoverflow-data-idf.json",lines=True)
# print schema
print("Schema:\n\n",df_idf.dtypes)
print("Number of questions,columns=",df_idf.shape)  Schema:
accepted_answer_id float64
answer_count int64
body object
comment_count int64
community_owned_date object
creation_date object
favorite_count float64
id int64
last_activity_date object
last_edit_date object
last_editor_display_name object
last_editor_user_id float64
owner_display_name object
owner_user_id float64
post_type_id int64
score int64
tags object
title object
view_count int64
dtype: object
Number of questions,columns= (20000, 19)
 Notice that this stack overflow dataset contains 19 fields including post title, body, tags, dates and other metadata which we don’t quite need for this tutorial. What we are mostly interested in for this tutorial, is the body and title which will become our source of text for keyword extraction. We will now create a field that combines both body and title so we have it in one field. We will also print the second text entry in our new field just to see what the text looks like.  
import re
def pre_process(text):
 
 # lowercase
 text=text.lower()
 
 #remove tags
 text=re.sub("","",text)
 
 # remove special characters and digits
 text=re.sub("(\\d|\\W)+"," ",text)
 
 return text
df_idf['text'] = df_idf['title'] + df_idf['body']
df_idf['text'] = df_idf['text'].apply(lambda x:pre_process(x))
#show the second 'text' just for fun
df_idf['text'][2]
    The text above is essentially a combination of the title and body of a stack overflow post. Hmmm, this doesn’t look very readable, does it? Well, that’s because we are cleaning the text after we concatenated the two fields (line 18). All of the cleaning happens in pre_process(..). You can do a lot more stuff in pre_process(..), such as eliminate all code sections, normalize the words to its root, etc, but for simplicity we perform only some mild pre-processing. Creating Vocabulary and Word Counts for IDF We now need to create the vocabulary and start the counting process. We can use the CountVectorizer to create a vocabulary from all the text in our df_idf['text'] followed by the counts of words in the vocabulary (see: usage examples for CountVectorizer).  
from sklearn.feature_extraction.text import CountVectorizer
import re
def get_stop_words(stop_file_path):
 """load stop words """
 
 with open(stop_file_path, 'r', encoding="utf-8") as f:
 stopwords = f.readlines()
 stop_set = set(m.strip() for m in stopwords)
 return frozenset(stop_set)
#load a set of stop words
stopwords=get_stop_words("resources/stopwords.txt")
#get the text column 
docs=df_idf['text'].tolist()
#create a vocabulary of words, 
#ignore words that appear in 85% of documents, 
#eliminate stop words
cv=CountVectorizer(max_df=0.85,stop_words=stopwords)
word_count_vector=cv.fit_transform(docs)
  While cv.fit(...) would only create the vocabulary, cv.fit_transform(...) creates the vocabulary and returns a term-document matrix which is what we want. With this, each column in the matrix represents a word in the vocabulary while each row represents the document in our dataset where the values in this case are the word counts. Note that with this representation, counts of some words could be 0 if the word did not appear in the corresponding document. Notice that in the code above, we are passing two parameters to CountVectorizer, max_df and stop_words. The first is just to say ignore all words that have appeared in 85% of the documents, since those may be unimportant. The later, is a custom stop words list. You can also use stop words that are native to sklearn by setting stop_words='english', but I personally find this to be quite limited. I used a custom stop word list used for this tutorial. The resulting shape of word_count_vector is (20000,124901) since we have 20,000 documents in our dataset (the rows) and the vocabulary size is 124,901. In some text mining applications such as clustering and text classification we typically limit the size of the vocabulary. It’s really easy to do this by setting max_features=vocab_size when instantiating CountVectorizer. For this tutorial let’s limit our vocabulary size to 10,000.  
cv=CountVectorizer(max_df=0.85,stop_words=stopwords,max_features=10000)
word_count_vector=cv.fit_transform(docs)
  Now, let’s look at 10 words from our vocabulary.  
list(cv.vocabulary_.keys())[:10]
  ['serializing',
 'private',
 'struct',
 'public',
 'class',
 'contains',
 'properties',
 'string',
 'serialize',
 'attempt']
 Sweet, these are mostly programming related. TfidfTransformer to Compute Inverse Document Frequency (IDF) It’s now time to compute the IDF values. In the code below, we are essentially taking the sparse matrix from CountVectorizer (word_count_vector) to generate the IDF when you invoke tfidf_transformer.fit(...)(see: basic usage example of tfidftransformer and tfidfvectorizer)  
from sklearn.feature_extraction.text import TfidfTransformer
tfidf_transformer=TfidfTransformer(smooth_idf=True,use_idf=True)
tfidf_transformer.fit(word_count_vector)
  An extremely important point to note here is that the IDF should always be based on a large corpora and should be representative of texts you would be using to extract keywords. This is why we are using texts from 20,000 stack overflow posts to compute the IDF instead of just a handful. I’ve seen several articles on the Web that compute the IDF using a handful of documents. You will defeat the whole purpose of IDF weighting if its not based on a large corpora as (a) your vocabulary becomes too small and (b) you have limited ability to observe the behavior of words that you do know about. Computing TF-IDF and Extracting Keywords Once we have our IDF computed, we are now ready to compute TF-IDF and then extract top keywords from the TF-IDF vectors. In this example, we will extract top keywords for the questions in data/stackoverflow-test.json. This data file has 500 questions with fields identical to that of data/stackoverflow-data-idf.json as we saw above. We will start by reading our test file, extracting the necessary fields (title and body) and getting the texts into a list.  
# read test docs into a dataframe and concatenate title and body
df_test=pd.read_json("data/stackoverflow-test.json",lines=True)
df_test['text'] = df_test['title'] + df_test['body']
df_test['text'] =df_test['text'].apply(lambda x:pre_process(x))
# get test docs into a list
docs_test=df_test['text'].tolist()
  The next step is to compute the tf-idf value for a given document in our test set by invoking tfidf_transformer.transform(...). This generates a vector of tf-idf scores. Next, we sort the words in the vector in descending order of tf-idf values and then iterate over to extract the top-n keywords. In the example below, we are extracting keywords for the first document in our test set.  
# you only needs to do this once, this is a mapping of index to 
feature_names=cv.get_feature_names()
# get the document that we want to extract keywords from
doc=docs_test[0]
#generate tf-idf for the given document
tf_idf_vector=tfidf_transformer.transform(cv.transform([doc]))
#sort the tf-idf vectors by descending order of scores
sorted_items=sort_coo(tf_idf_vector.tocoo())
#extract only the top n; n here is 10
keywords=extract_topn_from_vector(feature_names,sorted_items,10)
# now print the results
print("\n=====Doc=====")
print(doc)
print("\n===Keywords===")
for k in keywords:
 print(k,keywords[k])
   
def sort_coo(coo_matrix):
 tuples = zip(coo_matrix.col, coo_matrix.data)
 return sorted(tuples, key=lambda x: (x[1], x[0]), reverse=True)
def extract_topn_from_vector(feature_names, sorted_items, topn=10):
 """get the feature names and tf-idf score of top n items"""
 
 #use only topn items from vector
 sorted_items = sorted_items[:topn]
 score_vals = []
 feature_vals = []
 
 # word index and corresponding tf-idf score
 for idx, score in sorted_items:
 
 #keep track of feature name and its corresponding score
 score_vals.append(round(score, 3))
 feature_vals.append(feature_names[idx])
 #create a tuples of feature,score
 #results = zip(feature_vals,score_vals)
 results= {}
 for idx in range(len(feature_vals)):
 results[feature_vals[idx]]=score_vals[idx]
 
 return results
  The sort_coo(...) method essentially sorts the values in the vector while preserving the column index. Once you have the column index then its really easy to look-up the corresponding word value as you would see in extract_topn_from_vector(...) where we do feature_vals.append(feature_names[idx]). Example Results In this section, you will see some of the stack overflow questions followed by the top-10 keywords generated using the code above. Note that these questions are from the stackoverflow-test.json data file. Question about Eclipse Plugin integration   From the keywords above, the top keywords actually make sense, it talks about eclipse, maven, integrate, war and tomcat which are all unique to this specific question. There are a couple of keywords that could have been eliminated such as possibility and perhaps even project and you can further fine-tune what shows up on top by adding more common words to your stop list and you can even create your own set of stop list, very specific to your domain. Now let’s look at another example. Question about SQL Import   Even with all the html tags, because of the pre-processing, we are able to extract some pretty nice keywords here. The last word appropriately would qualify as a stop word. You can keep running different examples to get ideas of how to fine-tune the results. Wow! Now you can extract important keywords from any type of text! Some tips and tricks  You can easily save the resulting CountVectorizer and TfidfTransformer and load them back for use at a later time. Instead of using CountVectorizer followed by TfidfTransformer, you can directly use TfidfVectorizer by itself. This is equivalent to CountVectorizer followed by TfidfTransformer. In this example, we computed the tf-idf matrix for each document of interest and then extracted top terms from it. What you could also do is first applytfidf_transformer.transform(docs_test) which will generate a tf-idf matrix for all documents in docs_test at one go and then iterate over the resulting vectors to extract top keywords. The first approach is useful if you have one document coming in at a time. The second approach is more suitable when you want keywords from a fairly large set of documents.  See Also: How to correctly use TFIDFTransformer and TFIDFVectorizer?  Resources for python keyword extraction  Get the code samples for this tutorial  Explanation on using TfIdftransformer and Tfidfvectorizer  Stack overflow data on Google’s BigQuery
Python Keyword Extraction Tutorial using TF-IDF
https://kavita-ganesan.com/python-keyword-extraction/
https://github.com/sekwiatkowski/awesome-ai-services/blob/master/keyphrase-extraction/amazon-comprehend/amazon-comprehend-keyphrase-extraction.js
awesome-ai-services/keyphrase-extraction/amazon-comprehend/amazon-comprehend-keyphrase-extraction.js at master · sekwiatkowski/awesome-ai-services
2018-06-19
Sekwiatkowski
<div><div>
<nav>
<ul>
<li>
<div>
<div>
<ul>
<li>
<a href="https://github.com/features/actions">
<div>
<p>Actions</p><p>
 Automate any workflow
 </p></div>
</a></li>
<li>
<a href="https://github.com/features/packages">
<div>
<p>Packages</p><p>
 Host and manage packages
 </p></div>
</a></li>
<li>
<a href="https://github.com/features/security">
<div>
<p>Security</p><p>
 Find and fix vulnerabilities
 </p></div>
</a></li>
<li>
<a href="https://github.com/features/codespaces">
<div>
<p>Codespaces</p><p>
 Instant dev environments
 </p></div>
</a></li>
<li>
<a href="https://github.com/features/copilot">
<div>
<p>Copilot</p><p>
 Write better code with AI
 </p></div>
</a></li>
<li>
<a href="https://github.com/features/code-review">
<div>
<p>Code review</p><p>
 Manage code changes
 </p></div>
</a></li>
<li>
<a href="https://github.com/features/issues">
<div>
<p>Issues</p><p>
 Plan and track work
 </p></div>
</a></li>
<li>
<a href="https://github.com/features/discussions">
<div>
<p>Discussions</p><p>
 Collaborate outside of code
 </p></div>
</a></li>
</ul>
</div>
<div>
<p></p><ul>
<li>
<a href="https://github.com/features">
 All features
 
</a></li>
<li>
<a href="https://docs.github.com">
 Documentation
 
</a></li>
<li>
<a href="https://skills.github.com">
 GitHub Skills
 
</a></li>
<li>
<a href="https://github.blog">
 Blog
 
</a></li>
</ul>
</div>
</div>
</li>
<li>
<div>
<div>
<p></p><ul>
<li>
<a href="https://github.com/enterprise">
 Enterprise
 
</a></li>
<li>
<a href="https://github.com/team">
 Teams
 
</a></li>
<li>
<a href="https://github.com/enterprise/startups">
 Startups
 
</a></li>
<li>
<a href="https://education.github.com">
 Education
 
</a></li>
</ul>
</div>
<div>
<p></p><ul>
<li>
<a href="https://github.com/solutions/ci-cd">
 CI/CD &amp; Automation
 
</a></li>
<li>
<a href="https://github.com/solutions/devops">
 DevOps
 
</a></li>
<li>
<a href="https://resources.github.com/devops/fundamentals/devsecops">
 DevSecOps
 
</a></li>
</ul>
</div>
<div>
<p></p><ul>
<li>
<a href="https://resources.github.com/learn/pathways">
 Learning Pathways
 
</a></li>
<li>
<a href="https://resources.github.com">
 White papers, Ebooks, Webinars
 
</a></li>
<li>
<a href="https://github.com/customer-stories">
 Customer Stories
 
</a></li>
<li>
<a href="https://partner.github.com">
 Partners
 
</a></li>
</ul>
</div>
</div>
</li>
<li>
<div>
<div>
<ul>
<li>
<a href="https://github.com/sponsors">
<div>
<p>GitHub Sponsors</p><p>
 Fund open source developers
 </p></div>
</a></li>
</ul>
</div>
<div>
<ul>
<li>
<a href="https://github.com/readme">
<div>
<p>The ReadME Project</p><p>
 GitHub community articles
 </p></div>
</a></li>
</ul>
</div>
<div>
<p></p><ul>
<li>
<a href="https://github.com/topics">
 Topics
 
</a></li>
<li>
<a href="https://github.com/trending">
 Trending
 
</a></li>
<li>
<a href="https://github.com/collections">
 Collections
 
</a></li>
</ul>
</div>
</div>
</li>
<li>
<a href="https://github.com/pricing">Pricing</a>
</li>
</ul>
</nav>
<div>
<div>
<p><a href="https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsekwiatkowski%2Fawesome-ai-services%2Fblob%2Fmaster%2Fkeyphrase-extraction%2Famazon-comprehend%2Famazon-comprehend-keyphrase-extraction.js">
 Sign in
 </a>
</p></div>
<p><a href="https://github.com/signup?ref_cta=Sign+up&amp;ref_loc=header+logged+out&amp;ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fblob%2Fshow&amp;source=header-repo&amp;source_repo=sekwiatkowski%2Fawesome-ai-services">
 Sign up
 </a>
</p></div>
</div></div>||||I|||| Skip to content

 Navigation Menu

 Toggle navigation
 Sign in
 * Product
 + Actions
 Automate any workflow
 + Packages
 Host and manage packages
 + Security
 Find and fix vulnerabilities
 + Codespaces
 Instant dev environments
 + Copilot
 Write better code with AI
 + Code review
 Manage code changes
 + Issues
 Plan and track work
 + Discussions
 Collaborate outside of code
 Explore
 + All features
 + Documentation
 + GitHub Skills
 + Blog
 * Solutions
 For
 + Enterprise
 + Teams
 + Startups
 + Education
 By Solution
 + CI/CD & Automation
 + DevOps
 + DevSecOps
 Resources
 + Learning Pathways
 + White papers, Ebooks, Webinars
 + Customer Stories
 + Partners
 * Open Source
 + GitHub Sponsors
 Fund open source developers
 + The ReadME Project
 GitHub community articles
 Repositories
 + Topics
 + Trending
 + Collections
 * Pricing
 Search or jump to...

 Search code, repositories, users, issues, pull requests...

 Search
 Clear
 Search syntax tips

 Provide feedback

 We read every piece of feedback, and take your input very seriously.

 Include my email address so I can be contacted
 Cancel Submit feedback

 Saved searches

 Use saved searches to filter your results more quickly

 Name
 Query

 To see all available qualifiers, see our documentation.

 Cancel Create saved search
 Sign in
 Sign up
 You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert
 {{ message }}
 sekwiatkowski / awesome-ai-services Public
 * Notifications
 * Fork 23
 * Star 153
 * 
 * Code
 * Issues 0
 * Pull requests 0
 * Actions
 * Projects 0
 * Security
 * Insights
 Additional navigation options
 * Code
 * Issues
 * Pull requests
 * Actions
 * Projects
 * Security
 * Insights

 Footer

 © 2024 GitHub, Inc.

 Footer navigation

 * Terms
 * Privacy
 * Security
 * Status
 * Docs
 * Contact
 * Manage cookies
 * Do not share my personal information
 You can’t perform that action at this time.
Actions 
 Automate any workflow
  
 
Packages 
 Host and manage packages
  
 
Security 
 Find and fix vulnerabilities
  
 
Codespaces 
 Instant dev environments
  
 
Copilot 
 Write better code with AI
  
 
Code review 
 Manage code changes
  
 
Issues 
 Plan and track work
  
 
Discussions 
 Collaborate outside of code
  
 
  
 All features
 
 
 Documentation
 
 
 GitHub Skills
 
 
 Blog
 
 
  
 Enterprise
 
 
 Teams
 
 
 Startups
 
 
 Education
 
 
  
 CI/CD &amp; Automation
 
 
 DevOps
 
 
 DevSecOps
 
 
  
 Learning Pathways
 
 
 White papers, Ebooks, Webinars
 
 
 Customer Stories
 
 
 Partners
 
 
GitHub Sponsors 
 Fund open source developers
  
 
The ReadME Project 
 GitHub community articles
  
 
  
 Topics
 
 
 Trending
 
 
 Collections
 
 
Pricing
 
 Sign in
 
 
 
 Sign up
awesome-ai-services/keyphrase-extraction/amazon-comprehend/amazon-comprehend-keyphrase-extraction.js at master · sekwiatkowski/awesome-ai-services
https://github.com/sekwiatkowski/awesome-ai-services/blob/master/keyphrase-extraction/amazon-comprehend/amazon-comprehend-keyphrase-extraction.js
https://github.com/dongrixinyu/chinese_keyphrase_extractor
GitHub - dongrixinyu/chinese_keyphrase_extractor: An off-the-shelf tool for Chinese Keyphrase Extraction 一个快速从中文里抽取关键短语的工具，仅占35M内存 www.jionlp.com
2022-05-26
Dongrixinyu
<div><div>
 <article><h2>chinese_keyphrase_extractor (CKPE)</h2>
<p>一个从 <strong>中文自然语言文本</strong> 中抽取 <strong>关键短语</strong> 的工具，只消耗 <strong>35M</strong> 内存<br />
A tool for automatic <strong>keyphrase extraction</strong> from <strong>Chinese</strong> text.</p>
<h3>⭐<a href="http://www.jionlp.com/jionlp_online/extract_keyphrase">CKPE在线版</a>，可以直接进入试用关键短语抽取功能。</h3>
<h3>欢迎关注wx公众号“一个Bug”，获取该功能更多相关资讯，并可加入“NLP 交流群”</h3>
<h3>本项目已经<strong>ALREADY</strong>迁移至 <a href="https://github.com/dongrixinyu/JioNLP">JioNLP</a> 工具包，性能更好，速度更快哦~~~</h3>
<h3>当前工具语料最新时间为 2020年6月。此后的出现的新词不容易识别，须根据新语料处理。</h3>
<h2>应用场景 Application scenario</h2>
<h4>1.抽取关键短语</h4>
<ul>
<li>在很多关键词提取任务中，使用tfidf、textrank等方法提取得到的仅仅是若干零碎词汇。</li>
<li>这样的零碎词汇无法真正的表达文章的原本含义，我们并不想要它。</li>
<li>In many keyword extraction tasks, only a few fragmentary words are extracted when using tfidf, textrank and other methods.</li>
<li>Such fragmentary words cannot really express the original meaning of the article. We do not want it.<br />
例如：<br />
For example:</li>
</ul>
<div><pre><code>&gt;&gt;&gt; text = '朝鲜确认金正恩出访俄罗斯 将与普京举行会谈...'
&gt;&gt;&gt; keywords = ['俄罗斯', '朝鲜', '普京', '金正恩', '俄方']
</code></pre></div>
<ul>
<li>我们往往需要更细化的短语描述，来作为文本的关键信息展示。这样的需求在生成词云、提供摘要阅读、关键信息检索等任务中都非常重要。</li>
<li>We often need more detailed phrase descriptions to display the key information of the text. Such requirements, namely keyphrases extraction, are very important in generating word cloud, providing abstract reading, key information retrieval and other tasks.<br />
例如：
For example:</li>
</ul>
<div><pre><code>&gt;&gt;&gt; phrases = ['俄罗斯克里姆林宫', '邀请金正恩访俄', '最高司令官金正恩', 
 '朝方转交普京', '举行会谈']
</code></pre></div>
<h4>2.扩展相关短语词汇</h4>
<ul>
<li>有时产品和客户给定了一些词汇列表，比如化工经营业务词汇“聚氯乙烯”、“塑料”、“切割”、“金刚石”等。想要找到跟这些词汇相关的短语。</li>
<li>在做<strong>NER命名实体识别任务</strong>的时候，我们需要从文本中，将已有的类型词汇做扩充，如“机构”类别，但我们仅知道机构的一些特征，如常以“局”、“法院”、“办公室”等特征词结尾。</li>
<li>在下面的使用样例中，给出了上述两种需求的扩展短语识别的方法。</li>
</ul>
<h2>功能介绍 Function introduction</h2>
<p>为解决以上问题，基于北大分词器 pkuseg 工具，开发了一个关键短语抽取器，它可以方便地从文本中找出表达完成意思的关键短语。<br />
In order to solve the above problem, I developed a keyphrase extractor based on pkuseg segmentation tool, which can easily find out the keyphrases expressing the complete meaning from the text.</p>
<h2>使用方法 Usage</h2>
<h4>安装 Installation</h4>
<ul>
<li>仅支持 python3</li>
<li>自动安装 pkuseg 依赖包</li>
<li>python3 only supported</li>
<li>Automatic installation of pkuseg dependency package</li>
</ul>
<div><pre><code>git clone https://github.com/dongrixinyu/chinese_keyphrase_extractor
cd ./chinese_keyphrase_extractor
pip install .
</code></pre></div>
<ul>
<li><a href="https://github.com/dongrixinyu/JioNLP">JioNLP</a> 同样支持短语抽取，工具包安装方法：</li>
</ul>
<h4>示例代码 Sample code</h4>
<ul>
<li>输入必须为 <strong>utf-8</strong> 编码字符串</li>
<li>具体函数参数见代码</li>
<li>Input must be <strong>utf-8</strong> encoding string</li>
<li>Check code for specific function parameters</li>
</ul>
<h5>1.抽取关键短语</h5>
<div><pre><code>import ckpe 
ckpe_obj = ckpe.ckpe()
# 初次导入时会自动下载北大词性标注模型包，自动加载入内存（50M） 
# 若由于网速原因下载失败，请参考 https://github.com/lancopku/pkuseg-python 如何安装下载 pkuseg 默认模型 
# Speech Tagging Model Package of pkyseg will be downloaded automatically upon initial import 
# If downloading fails due to network speed, please refer to how to install and download pkuseg default model in https://github.com/lancopku/pkuseg-python 
text = '法国媒体最新披露，巴黎圣母院火灾当晚，第一次消防警报响起时，负责查验的保安找错了位置，因而可能贻误了救火的最佳时机。...'
key_phrases = ckpe_obj.extract_keyphrase(text)
print(key_phrases)
print(ckpe_obj.extract_keyphrase.__doc__)
</code></pre></div>
<ul>
<li><strong>JioNLP 工具包调用方法</strong>：</li>
</ul>
<div><pre><code>&gt;&gt;&gt; import jionlp as jio
&gt;&gt;&gt; text = '朝鲜确认金正恩出访俄罗斯 将与普京举行会谈...'
&gt;&gt;&gt; key_phrases = jio.keyphrase.extract_keyphrase(text)
&gt;&gt;&gt; print(key_phrases)
&gt;&gt;&gt; print(jio.keyphrase.extract_keyphrase.__doc__)
# ['俄罗斯克里姆林宫', '邀请金正恩访俄', '举行会谈',
# '朝方转交普京', '最高司令官金正恩']
</code></pre></div>
<h5>2.扩展类型短语</h5>
<div><pre><code>text = '聚氯乙烯树脂、塑料制品、切割工具、人造革、人造金刚石、农药（不含危险化学品）、针纺织品自产自销。...'
word_dict = {'聚氯乙烯': 1, '塑料': 1, '切割': 1, '金刚石': 1} # 词汇: 词频（词频若未知可全设 1）
key_phrases = ckpe_obj.extract_keyphrase(text, top_k=-1, specified_words=word_dict)
print(key_phrases)
</code></pre></div>
<h5>3.NER任务的短语扩充</h5>
<div><pre><code>text = '国务院下发通知，山西省法院、陕西省检察院、四川省法院、成都市教育局。...'
word_dict = {'局': 1, '国务院': 1, '检察院': 1, '法院': 1}
key_phrases = ckpe_obj.extract_keyphrase(text, top_k=-1, specified_words=word_dict, 
 remove_phrases_list=['麻将局'])
print(key_phrases)
</code></pre></div>
<h4><a href="https://github.com/dongrixinyu/chinese_keyphrase_extractor/wiki/%E5%85%B3%E9%94%AE%E7%9F%AD%E8%AF%AD%E6%8A%BD%E5%8F%96%E6%8A%80%E6%9C%AF%E7%AE%80%E8%BF%B0">关键短语抽取技术总结</a></h4>
<h4><a href="https://github.com/dongrixinyu/chinese_keyphrase_extractor/wiki/%E5%90%84%E4%B8%AA%E7%BB%9F%E8%AE%A1%E6%96%87%E4%BB%B6%E7%9A%84%E8%AE%A1%E7%AE%97%E6%96%B9%E6%B3%95">关于如何自己根据特定语料训练模型，各个文件的计算方法说明</a></h4>
<h4>计算主题向量</h4>
<ul>
<li>工具包中默认的主题模型参数由100万篇各个类型的新闻文本，以及少部分社交媒体文本训练得到。</li>
<li>若需要针对特定领域文本处理，则需要根据特定的语料重新训练模型，并按相应的文件格式做替换。</li>
<li>主题模型采用标准的 LDA 模型训练得到，工具包可选择 gensim、sklearn、familia 等，训练完毕后可以得到主题词的分布表示，进而可以得到每个词汇在不同主题下的分布。由此可以得出词汇的主题突出度。</li>
</ul>
<h2>新版 3.0 New Version 3.0</h2>
<ul>
<li>从 jieba 分词器迁移到 pkuseg，因为 jieba 分词器过于粗放</li>
<li>新增了停用词规则、虚词规则等</li>
<li>新增了 LDA 主题模型权重</li>
<li>Migrate from jieba segmenter to pkuseg because jieba segmenter is too extensive.</li>
<li>Stop words rule and function word rule have been added</li>
<li>Added LDA Topic Model Weight</li>
</ul>
<h2>原理 Principle of algorithm</h2>
<ul>
<li>首先基于 pkuseg 工具做分词和词性标注，再使用 tfidf 计算文本的关键词权重，</li>
<li>关键词提取算法找出碎片化的关键词，然后再根据相邻关键碎片词进行融合，重新计算权重，去除相似词汇。得到的融合的多个关键碎片即为关键短语。
<ul>
<li>短语的 token 长度不超过 12</li>
<li>短语中不可出现超过1个虚词</li>
<li>短语的两端 token 不可是虚词和停用词</li>
<li>短语中停用词数量不可以超过规定个数</li>
<li>短语重复度计算 MMR 添加其中</li>
<li>提供仅抽取名词短语功能</li>
</ul>
</li>
<li>使用预训练好的 LDA 模型，计算文本的主题概率分布，以及每一个候选短语的主题概率分布，得到最终权重</li>
<li>Firstly do word segmentation and part of speech tagging based on pkuseg tool, then use word level tfidf to calculate the keyword weight of the text.</li>
<li>Fuse the adjacent key fragment words, recalculates the weights, and removes the similar words. The fused key fragments are candidate keyphrases. Rules include:
<ul>
<li>Token length of phrase can not exceed 20</li>
<li>There cannot be more than one function word in a phrase</li>
<li>The token at both ends of the phrase should not be the function word and stop words</li>
<li>The number of stopwords in a phrase cannot exceed the specified number</li>
<li>Phrase repetition calculation is added</li>
<li>Provide only noun phrases extraction parameters</li>
</ul>
</li>
<li>Calculating the topic probability distribution of the text and the topic probability distribution of each candidate phrase by using the pre-trained LDA model to obtain the final weight</li>
</ul>
<h2>TODO</h2>
<ul>
<li>pkuseg 分词器造成的错误，如把时间识别为名词，数字识别为人名等</li>
<li>stopwords 表中一些词汇既可以做实词，又可以做虚词，如“本”，“类”，造成错误</li>
<li>规则过滤不完善造成的错误，或过滤过强造成的漏选</li>
</ul>
<h2>Reference</h2>
<ul>
<li>Teneva N , Cheng W . Salience Rank: Efficient Keyphrase Extraction with Topic Modeling[C]// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2017.</li>
<li>Liu Z , Huang W , Zheng Y , et al. Automatic Keyphrase Extraction via Topic Decomposition[C]// Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP 2010, 9-11 October 2010, MIT Stata Center, Massachusetts, USA, A meeting of SIGDAT, a Special Interest Group of the ACL. Association for Computational Linguistics, 2010.</li>
</ul>
<h2>我的窝 My blog</h2>
<p>如果觉得方便好用，可以请我喝杯咖啡 (●'◡'●) 也可以关注wx公众号“一个Bug”获取更多资料。</p>
<p><a href="/dongrixinyu/chinese_keyphrase_extractor/blob/master/payment_code.jpg"></a></p>
</article>
 </div>||||I|||| Skip to content
 Toggle navigation
 Sign up
 * Product
 + Actions
 Automate any workflow
 + Packages
 Host and manage packages
 + Security
 Find and fix vulnerabilities
 + Codespaces
 Instant dev environments
 + Copilot
 Write better code with AI
 + Code review
 Manage code changes
 + Issues
 Plan and track work
 + Discussions
 Collaborate outside of code
 + Explore
 + All features
 + Documentation
 + GitHub Skills
 + Blog
 * Solutions
 + For
 + Enterprise
 + Teams
 + Startups
 + Education
 + By Solution
 + CI/CD & Automation
 + DevOps
 + DevSecOps
 + Case Studies
 + Customer Stories
 + Resources
 * Open Source
 + GitHub Sponsors
 Fund open source developers
 + The ReadME Project
 GitHub community articles
 + Repositories
 + Topics
 + Trending
 + Collections
 * Pricing
 * In this repository All GitHub ↵
 Jump to ↵
 * No suggested jump to results
 * In this repository All GitHub ↵
 Jump to ↵
 * In this user All GitHub ↵
 Jump to ↵
 * In this repository All GitHub ↵
 Jump to ↵
 Sign in
 Sign up
 {{ message }}
 dongrixinyu / chinese_keyphrase_extractor Public
 * Notifications
 * Fork 61
 * Star 440

 An off-the-shelf tool for Chinese Keyphrase Extraction 一个快速从中文里抽取关键短语的工具，仅占35M内存 www.jionlp.com

 440 stars 61 forks
 Star
 Notifications
 * Code
 * Issues 2
 * Pull requests 0
 * Actions
 * Projects 0
 * Wiki
 * Security
 * Insights
 More
 * Code
 * Issues
 * Pull requests
 * Actions
 * Projects
 * Wiki
 * Security
 * Insights

 dongrixinyu/chinese_keyphrase_extractor

 This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
 master
 Switch branches/tags
 Branches Tags
 Could not load branches
 Nothing to show
 {{ refName }} default
 View all branches
 Could not load tags
 Nothing to show
 {{ refName }} default
 View all tags

 Name already in use

 A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
 Cancel Create
 1 branch 0 tags
 Code
 * Local
 * Codespaces
 * Clone
 HTTPS GitHub CLI

 Use Git or checkout with SVN using the web URL.

 Work fast with our official CLI. Learn more.

 * Open with GitHub Desktop
 * Download ZIP

 Sign In Required

 Please sign in to use Codespaces.

 Launching GitHub Desktop

 If nothing happens, download GitHub Desktop and try again.

 Launching GitHub Desktop

 If nothing happens, download GitHub Desktop and try again.

 Launching Xcode

 If nothing happens, download Xcode and try again.

 Launching Visual Studio Code

 Your codespace will open once ready.

 There was a problem preparing your codespace, please try again.

 Latest commit

 Git stats

 * 63 commits

 Files

 Permalink
 Failed to load latest commit information.
 Type
 Name
 Latest commit message
 Commit time
 ckpe
 Update chinese_key_phrases_extractor.py
 May 26, 2022 11:06
 MANIFEST.in
 fix manifest
 December 17, 2019 15:21
 README.md
 Update README.md
 July 28, 2022 11:00
 payment_code.jpg
 update code and qrcode image
 May 26, 2022 11:01
 setup.py
 add new function
 January 10, 2020 18:01
 View code
 chinese_keyphrase_extractor (CKPE) ⭐CKPE在线版，可以直接进入试用关键短语抽取功能。 欢迎关注wx公众号“一个Bug”，获取该功能更多相关资讯，并可加入“NLP 交流群” 本项目已经ALREADY迁移至 JioNLP 工具包，性能更好，速度更快哦~~~ 当前工具语料最新时间为 2020年6月。此后的出现的新词不容易识别，须根据新语料处理。 应用场景 Application scenario 1.抽取关键短语 2.扩展相关短语词汇 功能介绍 Function introduction 使用方法 Usage 安装 Installation 示例代码 Sample code 1.抽取关键短语 2.扩展类型短语 3.NER任务的短语扩充 关键短语抽取技术总结 关于如何自己根据特定语料训练模型，各个文件的计算方法说明 计算主题向量 新版 3.0 New Version 3.0 原理 Principle of algorithm TODO Reference 我的窝 My blog

 README.md

 chinese_keyphrase_extractor (CKPE)

 一个从 中文自然语言文本 中抽取 关键短语 的工具，只消耗 35M 内存
 A tool for automatic keyphrase extraction from Chinese text.

 ⭐CKPE在线版，可以直接进入试用关键短语抽取功能。

 欢迎关注wx公众号“一个Bug”，获取该功能更多相关资讯，并可加入“NLP 交流群”

 本项目已经ALREADY迁移至 JioNLP 工具包，性能更好，速度更快哦~~~

 当前工具语料最新时间为 2020年6月。此后的出现的新词不容易识别，须根据新语料处理。

 应用场景 Application scenario

 1.抽取关键短语

 * 在很多关键词提取任务中，使用tfidf、textrank等方法提取得到的仅仅是若干零碎词汇。
 * 这样的零碎词汇无法真正的表达文章的原本含义，我们并不想要它。
 * In many keyword extraction tasks, only a few fragmentary words are extracted when using tfidf, textrank and other methods.
 * Such fragmentary words cannot really express the original meaning of the article. We do not want it.
 例如：
 For example:
 >>> text = '朝鲜确认金正恩出访俄罗斯 将与普京举行会谈...'
 >>> keywords = ['俄罗斯', '朝鲜', '普京', '金正恩', '俄方']
 
 * 我们往往需要更细化的短语描述，来作为文本的关键信息展示。这样的需求在生成词云、提供摘要阅读、关键信息检索等任务中都非常重要。
 * We often need more detailed phrase descriptions to display the key information of the text. Such requirements, namely keyphrases extraction, are very important in generating word cloud, providing abstract reading, key information retrieval and other tasks.
 例如： For example:
 >>> phrases = ['俄罗斯克里姆林宫', '邀请金正恩访俄', '最高司令官金正恩', 
 '朝方转交普京', '举行会谈']
 

 2.扩展相关短语词汇

 * 有时产品和客户给定了一些词汇列表，比如化工经营业务词汇“聚氯乙烯”、“塑料”、“切割”、“金刚石”等。想要找到跟这些词汇相关的短语。
 * 在做NER命名实体识别任务的时候，我们需要从文本中，将已有的类型词汇做扩充，如“机构”类别，但我们仅知道机构的一些特征，如常以“局”、“法院”、“办公室”等特征词结尾。
 * 在下面的使用样例中，给出了上述两种需求的扩展短语识别的方法。

 功能介绍 Function introduction

 为解决以上问题，基于北大分词器 pkuseg 工具，开发了一个关键短语抽取器，它可以方便地从文本中找出表达完成意思的关键短语。
 In order to solve the above problem, I developed a keyphrase extractor based on pkuseg segmentation tool, which can easily find out the keyphrases expressing the complete meaning from the text.

 使用方法 Usage

 安装 Installation

 * 仅支持 python3
 * 自动安装 pkuseg 依赖包
 * python3 only supported
 * Automatic installation of pkuseg dependency package
 git clone https://github.com/dongrixinyu/chinese_keyphrase_extractor
 cd ./chinese_keyphrase_extractor
 pip install .
 
 * JioNLP 同样支持短语抽取，工具包安装方法：
 $ pip install jionlp
 

 示例代码 Sample code

 * 输入必须为 utf-8 编码字符串
 * 具体函数参数见代码
 * Input must be utf-8 encoding string
 * Check code for specific function parameters

 1.抽取关键短语

 import ckpe 
 
 ckpe_obj = ckpe.ckpe()
 # 初次导入时会自动下载北大词性标注模型包，自动加载入内存（50M） 
 # 若由于网速原因下载失败，请参考 https://github.com/lancopku/pkuseg-python 如何安装下载 pkuseg 默认模型 
 # Speech Tagging Model Package of pkyseg will be downloaded automatically upon initial import 
 # If downloading fails due to network speed, please refer to how to install and download pkuseg default model in https://github.com/lancopku/pkuseg-python 
 
 text = '法国媒体最新披露，巴黎圣母院火灾当晚，第一次消防警报响起时，负责查验的保安找错了位置，因而可能贻误了救火的最佳时机。...'
 key_phrases = ckpe_obj.extract_keyphrase(text)
 print(key_phrases)
 print(ckpe_obj.extract_keyphrase.__doc__)
 
 * JioNLP 工具包调用方法：
 >>> import jionlp as jio
 >>> text = '朝鲜确认金正恩出访俄罗斯 将与普京举行会谈...'
 >>> key_phrases = jio.keyphrase.extract_keyphrase(text)
 >>> print(key_phrases)
 >>> print(jio.keyphrase.extract_keyphrase.__doc__)
 
 # ['俄罗斯克里姆林宫', '邀请金正恩访俄', '举行会谈',
 # '朝方转交普京', '最高司令官金正恩']
 

 2.扩展类型短语

 text = '聚氯乙烯树脂、塑料制品、切割工具、人造革、人造金刚石、农药（不含危险化学品）、针纺织品自产自销。...'
 word_dict = {'聚氯乙烯': 1, '塑料': 1, '切割': 1, '金刚石': 1} # 词汇: 词频（词频若未知可全设 1）
 key_phrases = ckpe_obj.extract_keyphrase(text, top_k=-1, specified_words=word_dict)
 print(key_phrases)
 

 3.NER任务的短语扩充

 text = '国务院下发通知，山西省法院、陕西省检察院、四川省法院、成都市教育局。...'
 word_dict = {'局': 1, '国务院': 1, '检察院': 1, '法院': 1}
 key_phrases = ckpe_obj.extract_keyphrase(text, top_k=-1, specified_words=word_dict, 
 remove_phrases_list=['麻将局'])
 print(key_phrases)
 

 关键短语抽取技术总结

 关于如何自己根据特定语料训练模型，各个文件的计算方法说明

 计算主题向量

 * 工具包中默认的主题模型参数由100万篇各个类型的新闻文本，以及少部分社交媒体文本训练得到。
 * 若需要针对特定领域文本处理，则需要根据特定的语料重新训练模型，并按相应的文件格式做替换。
 * 主题模型采用标准的 LDA 模型训练得到，工具包可选择 gensim、sklearn、familia 等，训练完毕后可以得到主题词的分布表示，进而可以得到每个词汇在不同主题下的分布。由此可以得出词汇的主题突出度。

 新版 3.0 New Version 3.0

 * 从 jieba 分词器迁移到 pkuseg，因为 jieba 分词器过于粗放
 * 新增了停用词规则、虚词规则等
 * 新增了 LDA 主题模型权重
 * Migrate from jieba segmenter to pkuseg because jieba segmenter is too extensive.
 * Stop words rule and function word rule have been added
 * Added LDA Topic Model Weight

 原理 Principle of algorithm

 * 首先基于 pkuseg 工具做分词和词性标注，再使用 tfidf 计算文本的关键词权重，
 * 关键词提取算法找出碎片化的关键词，然后再根据相邻关键碎片词进行融合，重新计算权重，去除相似词汇。得到的融合的多个关键碎片即为关键短语。
 + 短语的 token 长度不超过 12
 + 短语中不可出现超过1个虚词
 + 短语的两端 token 不可是虚词和停用词
 + 短语中停用词数量不可以超过规定个数
 + 短语重复度计算 MMR 添加其中
 + 提供仅抽取名词短语功能
 * 使用预训练好的 LDA 模型，计算文本的主题概率分布，以及每一个候选短语的主题概率分布，得到最终权重
 * Firstly do word segmentation and part of speech tagging based on pkuseg tool, then use word level tfidf to calculate the keyword weight of the text.
 * Fuse the adjacent key fragment words, recalculates the weights, and removes the similar words. The fused key fragments are candidate keyphrases. Rules include:
 + Token length of phrase can not exceed 20
 + There cannot be more than one function word in a phrase
 + The token at both ends of the phrase should not be the function word and stop words
 + The number of stopwords in a phrase cannot exceed the specified number
 + Phrase repetition calculation is added
 + Provide only noun phrases extraction parameters
 * Calculating the topic probability distribution of the text and the topic probability distribution of each candidate phrase by using the pre-trained LDA model to obtain the final weight

 TODO

 * pkuseg 分词器造成的错误，如把时间识别为名词，数字识别为人名等
 * stopwords 表中一些词汇既可以做实词，又可以做虚词，如“本”，“类”，造成错误
 * 规则过滤不完善造成的错误，或过滤过强造成的漏选

 Reference

 * Teneva N , Cheng W . Salience Rank: Efficient Keyphrase Extraction with Topic Modeling[C]// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2017.
 * Liu Z , Huang W , Zheng Y , et al. Automatic Keyphrase Extraction via Topic Decomposition[C]// Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP 2010, 9-11 October 2010, MIT Stata Center, Massachusetts, USA, A meeting of SIGDAT, a Special Interest Group of the ACL. Association for Computational Linguistics, 2010.

 我的窝 My blog

 如果觉得方便好用，可以请我喝杯咖啡 (●'◡'●) 也可以关注wx公众号“一个Bug”获取更多资料。

 About

 An off-the-shelf tool for Chinese Keyphrase Extraction 一个快速从中文里抽取关键短语的工具，仅占35M内存 www.jionlp.com

 Topics

 topic-modeling keyphrase-extraction pkuseg

 Resources

 Readme

 Stars

 440 stars

 Watchers

 3 watching

 Forks

 61 forks

 Releases

 No releases published

 Packages 0

 No packages published

 Contributors 2

 * 
 * 

 Languages

 * Python 100.0%

 Footer

 © 2023 GitHub, Inc.

 Footer navigation

 * Terms
 * Privacy
 * Security
 * Status
 * Docs
 * Contact GitHub
 * Pricing
 * API
 * Training
 * Blog
 * About
 You can’t perform that action at this time.
 You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
chinese_keyphrase_extractor (CKPE)
一个从 中文自然语言文本 中抽取 关键短语 的工具，只消耗 35M 内存
A tool for automatic keyphrase extraction from Chinese text.
⭐CKPE在线版，可以直接进入试用关键短语抽取功能。
欢迎关注wx公众号“一个Bug”，获取该功能更多相关资讯，并可加入“NLP 交流群”
本项目已经ALREADY迁移至 JioNLP 工具包，性能更好，速度更快哦~~~
当前工具语料最新时间为 2020年6月。此后的出现的新词不容易识别，须根据新语料处理。
应用场景 Application scenario
1.抽取关键短语
在很多关键词提取任务中，使用tfidf、textrank等方法提取得到的仅仅是若干零碎词汇。
这样的零碎词汇无法真正的表达文章的原本含义，我们并不想要它。
In many keyword extraction tasks, only a few fragmentary words are extracted when using tfidf, textrank and other methods.
Such fragmentary words cannot really express the original meaning of the article. We do not want it.
例如：
For example:
  &gt;&gt;&gt; text = '朝鲜确认金正恩出访俄罗斯 将与普京举行会谈...'
&gt;&gt;&gt; keywords = ['俄罗斯', '朝鲜', '普京', '金正恩', '俄方']
  
我们往往需要更细化的短语描述，来作为文本的关键信息展示。这样的需求在生成词云、提供摘要阅读、关键信息检索等任务中都非常重要。
We often need more detailed phrase descriptions to display the key information of the text. Such requirements, namely keyphrases extraction, are very important in generating word cloud, providing abstract reading, key information retrieval and other tasks.
例如：
For example:
  &gt;&gt;&gt; phrases = ['俄罗斯克里姆林宫', '邀请金正恩访俄', '最高司令官金正恩', 
 '朝方转交普京', '举行会谈']
  
2.扩展相关短语词汇
有时产品和客户给定了一些词汇列表，比如化工经营业务词汇“聚氯乙烯”、“塑料”、“切割”、“金刚石”等。想要找到跟这些词汇相关的短语。
在做NER命名实体识别任务的时候，我们需要从文本中，将已有的类型词汇做扩充，如“机构”类别，但我们仅知道机构的一些特征，如常以“局”、“法院”、“办公室”等特征词结尾。
在下面的使用样例中，给出了上述两种需求的扩展短语识别的方法。
功能介绍 Function introduction
为解决以上问题，基于北大分词器 pkuseg 工具，开发了一个关键短语抽取器，它可以方便地从文本中找出表达完成意思的关键短语。
In order to solve the above problem, I developed a keyphrase extractor based on pkuseg segmentation tool, which can easily find out the keyphrases expressing the complete meaning from the text.
使用方法 Usage
安装 Installation
仅支持 python3
自动安装 pkuseg 依赖包
python3 only supported
Automatic installation of pkuseg dependency package
  git clone https://github.com/dongrixinyu/chinese_keyphrase_extractor
cd ./chinese_keyphrase_extractor
pip install .
  
 JioNLP 同样支持短语抽取，工具包安装方法：
示例代码 Sample code
输入必须为 utf-8 编码字符串
具体函数参数见代码
Input must be utf-8 encoding string
Check code for specific function parameters
1.抽取关键短语
  import ckpe 
ckpe_obj = ckpe.ckpe()
# 初次导入时会自动下载北大词性标注模型包，自动加载入内存（50M） 
# 若由于网速原因下载失败，请参考 https://github.com/lancopku/pkuseg-python 如何安装下载 pkuseg 默认模型 
# Speech Tagging Model Package of pkyseg will be downloaded automatically upon initial import 
# If downloading fails due to network speed, please refer to how to install and download pkuseg default model in https://github.com/lancopku/pkuseg-python 
text = '法国媒体最新披露，巴黎圣母院火灾当晚，第一次消防警报响起时，负责查验的保安找错了位置，因而可能贻误了救火的最佳时机。...'
key_phrases = ckpe_obj.extract_keyphrase(text)
print(key_phrases)
print(ckpe_obj.extract_keyphrase.__doc__)
  
 JioNLP 工具包调用方法：
  &gt;&gt;&gt; import jionlp as jio
&gt;&gt;&gt; text = '朝鲜确认金正恩出访俄罗斯 将与普京举行会谈...'
&gt;&gt;&gt; key_phrases = jio.keyphrase.extract_keyphrase(text)
&gt;&gt;&gt; print(key_phrases)
&gt;&gt;&gt; print(jio.keyphrase.extract_keyphrase.__doc__)
# ['俄罗斯克里姆林宫', '邀请金正恩访俄', '举行会谈',
# '朝方转交普京', '最高司令官金正恩']
  
2.扩展类型短语
  text = '聚氯乙烯树脂、塑料制品、切割工具、人造革、人造金刚石、农药（不含危险化学品）、针纺织品自产自销。...'
word_dict = {'聚氯乙烯': 1, '塑料': 1, '切割': 1, '金刚石': 1} # 词汇: 词频（词频若未知可全设 1）
key_phrases = ckpe_obj.extract_keyphrase(text, top_k=-1, specified_words=word_dict)
print(key_phrases)
  
3.NER任务的短语扩充
  text = '国务院下发通知，山西省法院、陕西省检察院、四川省法院、成都市教育局。...'
word_dict = {'局': 1, '国务院': 1, '检察院': 1, '法院': 1}
key_phrases = ckpe_obj.extract_keyphrase(text, top_k=-1, specified_words=word_dict, 
 remove_phrases_list=['麻将局'])
print(key_phrases)
  
 关键短语抽取技术总结 
 关于如何自己根据特定语料训练模型，各个文件的计算方法说明 
计算主题向量
工具包中默认的主题模型参数由100万篇各个类型的新闻文本，以及少部分社交媒体文本训练得到。
若需要针对特定领域文本处理，则需要根据特定的语料重新训练模型，并按相应的文件格式做替换。
主题模型采用标准的 LDA 模型训练得到，工具包可选择 gensim、sklearn、familia 等，训练完毕后可以得到主题词的分布表示，进而可以得到每个词汇在不同主题下的分布。由此可以得出词汇的主题突出度。
新版 3.0 New Version 3.0
从 jieba 分词器迁移到 pkuseg，因为 jieba 分词器过于粗放
新增了停用词规则、虚词规则等
新增了 LDA 主题模型权重
Migrate from jieba segmenter to pkuseg because jieba segmenter is too extensive.
Stop words rule and function word rule have been added
Added LDA Topic Model Weight
原理 Principle of algorithm
首先基于 pkuseg 工具做分词和词性标注，再使用 tfidf 计算文本的关键词权重，
关键词提取算法找出碎片化的关键词，然后再根据相邻关键碎片词进行融合，重新计算权重，去除相似词汇。得到的融合的多个关键碎片即为关键短语。
短语的 token 长度不超过 12
短语中不可出现超过1个虚词
短语的两端 token 不可是虚词和停用词
短语中停用词数量不可以超过规定个数
短语重复度计算 MMR 添加其中
提供仅抽取名词短语功能
使用预训练好的 LDA 模型，计算文本的主题概率分布，以及每一个候选短语的主题概率分布，得到最终权重
Firstly do word segmentation and part of speech tagging based on pkuseg tool, then use word level tfidf to calculate the keyword weight of the text.
Fuse the adjacent key fragment words, recalculates the weights, and removes the similar words. The fused key fragments are candidate keyphrases. Rules include:
Token length of phrase can not exceed 20
There cannot be more than one function word in a phrase
The token at both ends of the phrase should not be the function word and stop words
The number of stopwords in a phrase cannot exceed the specified number
Phrase repetition calculation is added
Provide only noun phrases extraction parameters
Calculating the topic probability distribution of the text and the topic probability distribution of each candidate phrase by using the pre-trained LDA model to obtain the final weight
TODO
pkuseg 分词器造成的错误，如把时间识别为名词，数字识别为人名等
stopwords 表中一些词汇既可以做实词，又可以做虚词，如“本”，“类”，造成错误
规则过滤不完善造成的错误，或过滤过强造成的漏选
Reference
Teneva N , Cheng W . Salience Rank: Efficient Keyphrase Extraction with Topic Modeling[C]// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2017.
Liu Z , Huang W , Zheng Y , et al. Automatic Keyphrase Extraction via Topic Decomposition[C]// Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP 2010, 9-11 October 2010, MIT Stata Center, Massachusetts, USA, A meeting of SIGDAT, a Special Interest Group of the ACL. Association for Computational Linguistics, 2010.
我的窝 My blog
如果觉得方便好用，可以请我喝杯咖啡 (●'◡'●) 也可以关注wx公众号“一个Bug”获取更多资料。
GitHub - dongrixinyu/chinese_keyphrase_extractor: An off-the-shelf tool for Chinese Keyphrase Extraction 一个快速从中文里抽取关键短语的工具，仅占35M内存 www.jionlp.com
https://github.com/dongrixinyu/chinese_keyphrase_extractor