jchevallard commited on
Commit
03f42ec
·
verified ·
1 Parent(s): 96a2960

Upload folder using huggingface_hub

Browse files
Files changed (3) hide show
  1. README.md +32 -0
  2. tokenizer.json +0 -0
  3. tokenizer_config.json +9 -0
README.md ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ tags:
4
+ - transformers.js
5
+ - tokenizers
6
+ license: mit
7
+ ---
8
+
9
+ Cloned from [Xenova/claude-tokenizer](https://huggingface.co/Xenova/claude-tokenizer)
10
+
11
+
12
+ # Claude Tokenizer
13
+
14
+ A 🤗-compatible version of the **Claude tokenizer** (adapted from [anthropics/anthropic-sdk-python](https://github.com/anthropics/anthropic-sdk-python)). This means it can be used with Hugging Face libraries including [Transformers](https://github.com/huggingface/transformers), [Tokenizers](https://github.com/huggingface/tokenizers), and [Transformers.js](https://github.com/xenova/transformers.js).
15
+
16
+ ## Example usage:
17
+
18
+ ### Transformers/Tokenizers
19
+ ```py
20
+ from transformers import GPT2TokenizerFast
21
+
22
+ tokenizer = GPT2TokenizerFast.from_pretrained('Xenova/claude-tokenizer')
23
+ assert tokenizer.encode('hello world') == [9381, 2253]
24
+ ```
25
+
26
+ ### Transformers.js
27
+ ```js
28
+ import { AutoTokenizer } from '@xenova/transformers';
29
+
30
+ const tokenizer = await AutoTokenizer.from_pretrained('Xenova/claude-tokenizer');
31
+ const tokens = tokenizer.encode('hello world'); // [9381, 2253]
32
+ ```
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "bos_token": "<EOT>",
4
+ "clean_up_tokenization_spaces": true,
5
+ "eos_token": "<EOT>",
6
+ "model_max_length": 200000,
7
+ "tokenizer_class": "GPT2TokenizerFast",
8
+ "unk_token": "<EOT>"
9
+ }