{"action":"created","issue":{"url":"https://api.github.com/repos/google/sentencepiece/issues/181","repository_url":"https://api.github.com/repos/google/sentencepiece","labels_url":"https://api.github.com/repos/google/sentencepiece/issues/181/labels{/name}","comments_url":"https://api.github.com/repos/google/sentencepiece/issues/181/comments","events_url":"https://api.github.com/repos/google/sentencepiece/issues/181/events","html_url":"https://github.com/google/sentencepiece/issues/181","id":351492568,"node_id":"MDU6SXNzdWUzNTE0OTI1Njg=","number":181,"title":"Frequency weighted training sentences","user":{"login":"feddyfedfed","id":36654108,"node_id":"MDQ6VXNlcjM2NjU0MTA4","avatar_url":"https://avatars0.githubusercontent.com/u/36654108?v=4","gravatar_id":"","url":"https://api.github.com/users/feddyfedfed","html_url":"https://github.com/feddyfedfed","followers_url":"https://api.github.com/users/feddyfedfed/followers","following_url":"https://api.github.com/users/feddyfedfed/following{/other_user}","gists_url":"https://api.github.com/users/feddyfedfed/gists{/gist_id}","starred_url":"https://api.github.com/users/feddyfedfed/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/feddyfedfed/subscriptions","organizations_url":"https://api.github.com/users/feddyfedfed/orgs","repos_url":"https://api.github.com/users/feddyfedfed/repos","events_url":"https://api.github.com/users/feddyfedfed/events{/privacy}","received_events_url":"https://api.github.com/users/feddyfedfed/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-17T07:49:34Z","updated_at":"2018-08-17T12:32:52Z","closed_at":null,"author_association":"NONE","body":"I don't know if this is already implemented or if there's a workaround for this.  Sometimes, due to the large amount of data, what we have are training sentences that are already uniquely condensed (and we just keep their frequency counts somewhere).  Of course, this condensed training set does not necessarily reflect the original statistics of the occurrences of each word and may not reflect the optimal choices for the tokens.\r\n\r\nThrough experiments, I create two separate segmentation models using the (1) 10M most frequent sentences (unique), and (2) 10M random sentences from the entire set (duplicates are spread in the chronological order that they were generated).   Case (2) is of course favored by the empirical evidence suggested by the developers.  However, for our task I found that case (1) model is performing better.  So the experiment was inconclusive in deciding on which seeds set to choose next time.  \r\n\r\nI also found that the maximum number of training set is 100M and I wanted to take advantage of this limit since we have a huge amount of data.  Of course I could simply generate random 100M sentences from all the training data, but I found it quite empirical.  Especially since we're working on CJK languages.\r\n\r\nSo I guess in a nutshell:\r\n(1) Is there a way to incorporate frequency counts of sentences?\r\n(2) What is a statistically reliable way to condense a huge amount of data (let's say the original size is 300% of the maximum allowed), but still reflect the (approximate) statistics of the words?\r\n"},"comment":{"url":"https://api.github.com/repos/google/sentencepiece/issues/comments/413851180","html_url":"https://github.com/google/sentencepiece/issues/181#issuecomment-413851180","issue_url":"https://api.github.com/repos/google/sentencepiece/issues/181","id":413851180,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMzg1MTE4MA==","user":{"login":"taku910","id":11470708,"node_id":"MDQ6VXNlcjExNDcwNzA4","avatar_url":"https://avatars3.githubusercontent.com/u/11470708?v=4","gravatar_id":"","url":"https://api.github.com/users/taku910","html_url":"https://github.com/taku910","followers_url":"https://api.github.com/users/taku910/followers","following_url":"https://api.github.com/users/taku910/following{/other_user}","gists_url":"https://api.github.com/users/taku910/gists{/gist_id}","starred_url":"https://api.github.com/users/taku910/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/taku910/subscriptions","organizations_url":"https://api.github.com/users/taku910/orgs","repos_url":"https://api.github.com/users/taku910/repos","events_url":"https://api.github.com/users/taku910/events{/privacy}","received_events_url":"https://api.github.com/users/taku910/received_events","type":"User","site_admin":false},"created_at":"2018-08-17T12:32:52Z","updated_at":"2018-08-17T12:32:52Z","author_association":"COLLABORATOR","body":"Thank you for using SentencePiece.\r\n\r\nThough it is not documented, `spm_train --input=tsv_file --input_format=tsv ..`  allows for loading a corpus in tsv format where the first column is the sentence and the second column is frequency.  `--input_format=tsv` is originally prepared to train the model from the word list with frequency.\r\n\r\nAs for the training data, I feel that the default size 100M is enough for SPM training as long as they are sampled appropriately from the full corpus.  SentencePiece adopts a simple uni-gram language model with relatively smaller vocab size.  We would be able to get a reliable statistics only from the small subset of the corpus.  However, it would be nicer to check the output segmentation changing the size of corpus. \r\n\r\n\r\n\r\n"}}
{"actor":{"display_login":"taku910"}}