metadata

title: wsws-chatbot
emoji: 📊
colorFrom: green
colorTo: indigo
sdk: gradio
sdk_version: 5.12.0
python_version: 3.12
app_file: article_embedding/app/ui.py
pinned: false
short_description: Ask the WSWS chatbot

Filter

Identify the top 60 most important articles of the year

Variables	Description
articlesPerYear	Number of articles per year
perspectivePercent	Percentage of perspectives
usPercent	Percentage of US articles
maxArticlesPerAuthor	Maximum number of articles per author

Important topics

Topic ID	Topic Name
3f1db5b4-7915-4cef-a132-3b2e51f0728d	US Politics
105b9ec2-0cdd-480d-9c55-6f806fa7b9bc	Perspectives
04d2a008-8cef-4928-9a38-86b205a7590f	The coronavirus pandemic
ea69254c-a976-4e67-a79c-843202a10e55	United States
b0e983c5-6e85-4829-a9fe-3c25f02c211f	North America

Analysis

Document metadata structure

The metadata is stored as JSON
Yes, there are structured attributes in the metadata
There are many other structured attributes but none that are relevant to the system design

Vector database setup

Qdrant
Documents are embedded using the Stella 400M v5 embedding

Integration of filters and constraints

Excellent question. The constraints need not be strict. They should be approximately met, since relevance to the provided queries is also important
The documents should be selected first based on relevance and then adjust the final selection to meet the constraints. It's ok to select less relevant articles to get closer to the constraints, but up to a certain point.

Date range filtering

The date range is strict

Result volume and ranking logic

Great question too. The articles should be ranked based on their relevance, with all queries adding documents to the result set, but choosing the most relevant ones from this pool. For example, it would be ok if all articles end up being derived from only one query.

Combining multiple queries

The former. Each query is run separately and then the results are combined based on the document relevance score. Since the same document may be returned for different queries, in each case with a different relevance score, the highest relevance score should be used.

User preferences and constraint handling

Exactly, the result set should be post processed to meet constraints

Performance considerations

The document database has about 100,000 documents
No latency requirements. The accuracy of the system is the only requirement

Evaluation and Benchmarks

The system success will be measured based on relevance judgments and coverage of constraints, in that order
A feedback loop to improve the retrieval strategies or embedding approach is definitely on the table, if needed.
```
  self.embedding_model = SentenceTransformerModel("dunzhang/stella_en_400M_v5")
```

Performance benchmarks

The script at article_embedding/modal_app.py was used to measure the performance of the embedding model. The results are as follows:

GPU	Speed	Cost
T4	130.0ms	2.13¢
L4	82.7ms	1.84¢
M3 Max	60.0ms	N/A
A10G	64.2ms	1.96¢
A100	42.2ms	3.26¢
H100	24.9ms	3.15¢

These are benchmarks on the M3 Max 128GB:

Model	f32	f16	Context	Size
jasper sdpa	373ms	314ms	1,024	1,024
jasper	354ms	330ms	1,024	1,024
stella-en-400M	65ms	62ms	131,072	1,024

Models

Jasper

Issues

$  cd /Users/jlopez/git-repos/icfi/article-embedding ; /usr/bin/env /Users/jlopez/git-repos/icfi/article-embedding/.venv/bin/python /Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher 63791 -- article_embedding/loader.py
  0%|                                                                                                                                                                               | 0/314855 [00:00<?, ?it/s]E+00379.644: /handling disconnect from Adapter/
             Failed to kill Debuggee[PID=93387]

             Traceback (most recent call last):
               File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/launcher/debuggee.py", line 190, in kill
                 os.killpg(process.pid, signal.SIGKILL)
             PermissionError: [Errno 1] Operation not permitted

             Stack where logged:
               File "/Users/jlopez/.pyenv/versions/3.12.5/lib/python3.12/threading.py", line 1032, in _bootstrap
                 self._bootstrap_inner()
               File "/Users/jlopez/.pyenv/versions/3.12.5/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
                 self.run()
               File "/Users/jlopez/.pyenv/versions/3.12.5/lib/python3.12/threading.py", line 1012, in run
                 self._target(*self._args, **self._kwargs)
               File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/common/messaging.py", line 1458, in _run_handlers
                 handler()
               File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/common/messaging.py", line 1488, in _handle_disconnect
                 handler()
               File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/launcher/handlers.py", line 152, in disconnect
                 debuggee.kill()
               File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/launcher/debuggee.py", line 192, in kill
                 log.swallow_exception("Failed to kill {0}", describe())
               File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/common/log.py", line 215, in swallow_exception
                 _exception(format_string, *args, **kwargs)


E+00379.648: Failed to kill Debuggee[PID=93387]

             Traceback (most recent call last):
               File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/launcher/debuggee.py", line 190, in kill
                 os.killpg(process.pid, signal.SIGKILL)
             PermissionError: [Errno 1] Operation not permitted

             Stack where logged:
               File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/launcher/debuggee.py", line 192, in kill
                 log.swallow_exception("Failed to kill {0}", describe())
               File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/common/log.py", line 215, in swallow_exception
                 _exception(format_string, *args, **kwargs)