prowriting commited on
Commit
d65d045
Β·
verified Β·
1 Parent(s): 40fcf94

Fix tokenizer build errors by installing SentencePiece + tiktoken

Browse files

- Added `sentencepiece` and `tiktoken` to requirements.txt to resolve Hugging Face Transformers tokenizer errors.
- These libraries are required for T5 and other models that rely on SentencePiece or tiktoken-based vocabularies.
- Updated Docker build to ensure that both dependencies are preinstalled before loading models.
- This prevents runtime failures such as:
ValueError: Converting from SentencePiece and Tiktoken failed
or
ValueError: `tiktoken` is required to read a `tiktoken` file.

Files changed (3) hide show
  1. README.md +6 -26
  2. app.py +7 -0
  3. requirements.txt +3 -1
README.md CHANGED
@@ -1,34 +1,14 @@
1
  ---
2
- title: Paraphraser API
3
- emoji: ✨
4
- colorFrom: indigo
5
  colorTo: green
6
  sdk: gradio
7
  sdk_version: "4.29.0"
8
- app_file: main.py
9
  pinned: false
10
  ---
11
 
12
- # Paraphraser API
13
 
14
- A minimal Hugging Face Space using Gradio and Transformers to paraphrase text.
15
-
16
- Enter a sentence in the UI and receive a paraphrased version instantly.
17
-
18
- ---
19
-
20
- ## πŸš€ Example Usage
21
-
22
- **Input:**
23
- ```
24
- The quick brown fox jumps over the lazy dog.
25
- ```
26
-
27
- **Output (paraphrased):**
28
- ```
29
- A fast brown fox leaps across a sleeping dog.
30
- ```
31
-
32
- ---
33
-
34
- πŸ”Ή Try it out using the text box below!
 
1
  ---
2
+ title: My Hugging Face Space
3
+ emoji: πŸš€
4
+ colorFrom: blue
5
  colorTo: green
6
  sdk: gradio
7
  sdk_version: "4.29.0"
8
+ app_file: app.py
9
  pinned: false
10
  ---
11
 
12
+ # My Hugging Face Space
13
 
14
+ This is a demo space fixed with proper configuration.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
app.py ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ import gradio as gr
2
+
3
+ def greet(name):
4
+ return f"Hello {name}!"
5
+
6
+ iface = gr.Interface(fn=greet, inputs="text", outputs="text")
7
+ iface.launch()
requirements.txt CHANGED
@@ -1,3 +1,5 @@
 
1
  transformers
2
  torch
3
- gradio
 
 
1
+ gradio==4.29.0
2
  transformers
3
  torch
4
+ sentencepiece
5
+ tiktoken