fix: Adjust call to generate_replacements to use token IDs for prefix_tokens 98f1760 mebubo commited on Oct 6, 2024
feat: Update generate_replacements to accept prefix as token list 76e131a mebubo commited on Oct 6, 2024
fix: Include attention mask in model input and function parameters b47d499 mebubo commited on Oct 6, 2024
fix: Set attention mask in model inputs to avoid unexpected behavior 9029ade mebubo commited on Oct 6, 2024
feat: Add tqdm for progress tracking and time measurement in loop 1f2d72c mebubo commited on Oct 6, 2024
refactor: Update type hints to use Python 3.12 style in generate_replacements 15b7594 mebubo commited on Oct 6, 2024
refactor: Improve efficiency of generate_replacements by reducing model calls 91515a1 mebubo commited on Oct 6, 2024
refactor: Simplify main function by removing unnecessary indentation ada166c mebubo commited on Oct 6, 2024
fix: Pass tokenizer to calculate_log_probabilities function to resolve error c6407ad mebubo commited on Oct 5, 2024
refactor: Modularize code by creating functions for model loading and processing 8e36e52 mebubo commited on Oct 5, 2024
feat: add filtering for low probability words based on log probability threshold b2b409d mebubo commited on Oct 4, 2024
refactor: separate text processing logic into a new module for better organization 6735ae4 mebubo commited on Oct 4, 2024
mebubo
commited on
