Spaces:
Sleeping
Sleeping
| title: wsws-chatbot | |
| emoji: 📊 | |
| colorFrom: green | |
| colorTo: indigo | |
| sdk: gradio | |
| sdk_version: 5.12.0 | |
| python_version: 3.12 | |
| app_file: article_embedding/app/ui.py | |
| pinned: false | |
| short_description: Ask the WSWS chatbot | |
| ## Filter | |
| Identify the top 60 most important articles of the year | |
| | Variables | Description | | |
| | -------------------- | ------------------------------------- | | |
| | articlesPerYear | Number of articles per year | | |
| | perspectivePercent | Percentage of perspectives | | |
| | usPercent | Percentage of US articles | | |
| | maxArticlesPerAuthor | Maximum number of articles per author | | |
| ### Important topics | |
| | Topic ID | Topic Name | | |
| | ------------------------------------ | ------------------------ | | |
| | 3f1db5b4-7915-4cef-a132-3b2e51f0728d | US Politics | | |
| | 105b9ec2-0cdd-480d-9c55-6f806fa7b9bc | Perspectives | | |
| | 04d2a008-8cef-4928-9a38-86b205a7590f | The coronavirus pandemic | | |
| | ea69254c-a976-4e67-a79c-843202a10e55 | United States | | |
| | b0e983c5-6e85-4829-a9fe-3c25f02c211f | North America | | |
| ## Analysis | |
| 1. Document metadata structure | |
| - The metadata is stored as JSON | |
| - Yes, there are structured attributes in the metadata | |
| - There are many other structured attributes but none that are relevant to the system design | |
| 2. Vector database setup | |
| - Qdrant | |
| - Documents are embedded using the Stella 400M v5 embedding | |
| 3. Integration of filters and constraints | |
| - Excellent question. The constraints need not be strict. They should be approximately met, since relevance to the provided queries is also important | |
| - The documents should be selected first based on relevance and then adjust the final selection to meet the constraints. It's ok to select less relevant articles to get closer to the constraints, but up to a certain point. | |
| 4. Date range filtering | |
| - The date range is strict | |
| 5. Result volume and ranking logic | |
| - Great question too. The articles should be ranked based on their relevance, with all queries adding documents to the result set, but choosing the most relevant ones from this pool. For example, it would be ok if all articles end up being derived from only one query. | |
| 6. Combining multiple queries | |
| - The former. Each query is run separately and then the results are combined based on the document relevance score. Since the same document may be returned for different queries, in each case with a different relevance score, the highest relevance score should be used. | |
| 7. User preferences and constraint handling | |
| - Exactly, the result set should be post processed to meet constraints | |
| 8. Performance considerations | |
| - The document database has about 100,000 documents | |
| - No latency requirements. The accuracy of the system is the only requirement | |
| 9. Evaluation and Benchmarks | |
| - The system success will be measured based on relevance judgments and coverage of constraints, in that order | |
| - A feedback loop to improve the retrieval strategies or embedding approach is definitely on the table, if needed. | |
| self.embedding_model = SentenceTransformerModel("dunzhang/stella_en_400M_v5") | |
| # Performance benchmarks | |
| The script at `article_embedding/modal_app.py` was used to measure the performance of the embedding model. The results are as follows: | |
| | GPU | Speed | Cost | | |
| | :----: | ------: | ----: | | |
| | T4 | 130.0ms | 2.13¢ | | |
| | L4 | 82.7ms | 1.84¢ | | |
| | M3 Max | 60.0ms | N/A | | |
| | A10G | 64.2ms | 1.96¢ | | |
| | A100 | 42.2ms | 3.26¢ | | |
| | H100 | 24.9ms | 3.15¢ | | |
| These are benchmarks on the M3 Max 128GB: | |
| | Model | f32 | f16 | Context | Size | | |
| | -------------- | ----: | ----: | ------: | ----: | | |
| | jasper sdpa | 373ms | 314ms | 1,024 | 1,024 | | |
| | jasper | 354ms | 330ms | 1,024 | 1,024 | | |
| | stella-en-400M | 65ms | 62ms | 131,072 | 1,024 | | |
| ## Models | |
| [Jasper](https://huggingface.co/infgrad/jasper_en_vision_language_v1) | |
| ## Issues | |
| ```sh | |
| $ cd /Users/jlopez/git-repos/icfi/article-embedding ; /usr/bin/env /Users/jlopez/git-repos/icfi/article-embedding/.venv/bin/python /Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher 63791 -- article_embedding/loader.py | |
| 0%| | 0/314855 [00:00<?, ?it/s]E+00379.644: /handling disconnect from Adapter/ | |
| Failed to kill Debuggee[PID=93387] | |
| Traceback (most recent call last): | |
| File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/launcher/debuggee.py", line 190, in kill | |
| os.killpg(process.pid, signal.SIGKILL) | |
| PermissionError: [Errno 1] Operation not permitted | |
| Stack where logged: | |
| File "/Users/jlopez/.pyenv/versions/3.12.5/lib/python3.12/threading.py", line 1032, in _bootstrap | |
| self._bootstrap_inner() | |
| File "/Users/jlopez/.pyenv/versions/3.12.5/lib/python3.12/threading.py", line 1075, in _bootstrap_inner | |
| self.run() | |
| File "/Users/jlopez/.pyenv/versions/3.12.5/lib/python3.12/threading.py", line 1012, in run | |
| self._target(*self._args, **self._kwargs) | |
| File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/common/messaging.py", line 1458, in _run_handlers | |
| handler() | |
| File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/common/messaging.py", line 1488, in _handle_disconnect | |
| handler() | |
| File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/launcher/handlers.py", line 152, in disconnect | |
| debuggee.kill() | |
| File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/launcher/debuggee.py", line 192, in kill | |
| log.swallow_exception("Failed to kill {0}", describe()) | |
| File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/common/log.py", line 215, in swallow_exception | |
| _exception(format_string, *args, **kwargs) | |
| E+00379.648: Failed to kill Debuggee[PID=93387] | |
| Traceback (most recent call last): | |
| File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/launcher/debuggee.py", line 190, in kill | |
| os.killpg(process.pid, signal.SIGKILL) | |
| PermissionError: [Errno 1] Operation not permitted | |
| Stack where logged: | |
| File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/launcher/debuggee.py", line 192, in kill | |
| log.swallow_exception("Failed to kill {0}", describe()) | |
| File "/Users/jlopez/.vscode/extensions/ms-python.debugpy-2024.10.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/common/log.py", line 215, in swallow_exception | |
| _exception(format_string, *args, **kwargs) | |
| ``` | |