{"action":"created","issue":{"url":"https://api.github.com/repos/AbrahamSanders/seq2seq-chatbot/issues/2","repository_url":"https://api.github.com/repos/AbrahamSanders/seq2seq-chatbot","labels_url":"https://api.github.com/repos/AbrahamSanders/seq2seq-chatbot/issues/2/labels{/name}","comments_url":"https://api.github.com/repos/AbrahamSanders/seq2seq-chatbot/issues/2/comments","events_url":"https://api.github.com/repos/AbrahamSanders/seq2seq-chatbot/issues/2/events","html_url":"https://github.com/AbrahamSanders/seq2seq-chatbot/issues/2","id":323143841,"number":2,"title":"Cloud Training and Learning","user":{"login":"sagarhingal","id":16017053,"avatar_url":"https://avatars1.githubusercontent.com/u/16017053?v=4","gravatar_id":"","url":"https://api.github.com/users/sagarhingal","html_url":"https://github.com/sagarhingal","followers_url":"https://api.github.com/users/sagarhingal/followers","following_url":"https://api.github.com/users/sagarhingal/following{/other_user}","gists_url":"https://api.github.com/users/sagarhingal/gists{/gist_id}","starred_url":"https://api.github.com/users/sagarhingal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sagarhingal/subscriptions","organizations_url":"https://api.github.com/users/sagarhingal/orgs","repos_url":"https://api.github.com/users/sagarhingal/repos","events_url":"https://api.github.com/users/sagarhingal/events{/privacy}","received_events_url":"https://api.github.com/users/sagarhingal/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-05-15T09:47:28Z","updated_at":"2018-05-21T18:13:18Z","closed_at":null,"author_association":"NONE","body":"1. How can i deploy this code into the cloud to train faster, like google's compute engine or something?\r\n2. Have you thought of implementing on-the-go learning? Like updating the weights after each session? If not, any thoughts on that?"},"comment":{"url":"https://api.github.com/repos/AbrahamSanders/seq2seq-chatbot/issues/comments/390736931","html_url":"https://github.com/AbrahamSanders/seq2seq-chatbot/issues/2#issuecomment-390736931","issue_url":"https://api.github.com/repos/AbrahamSanders/seq2seq-chatbot/issues/2","id":390736931,"user":{"login":"AbrahamSanders","id":29312744,"avatar_url":"https://avatars3.githubusercontent.com/u/29312744?v=4","gravatar_id":"","url":"https://api.github.com/users/AbrahamSanders","html_url":"https://github.com/AbrahamSanders","followers_url":"https://api.github.com/users/AbrahamSanders/followers","following_url":"https://api.github.com/users/AbrahamSanders/following{/other_user}","gists_url":"https://api.github.com/users/AbrahamSanders/gists{/gist_id}","starred_url":"https://api.github.com/users/AbrahamSanders/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AbrahamSanders/subscriptions","organizations_url":"https://api.github.com/users/AbrahamSanders/orgs","repos_url":"https://api.github.com/users/AbrahamSanders/repos","events_url":"https://api.github.com/users/AbrahamSanders/events{/privacy}","received_events_url":"https://api.github.com/users/AbrahamSanders/received_events","type":"User","site_admin":false},"created_at":"2018-05-21T18:13:18Z","updated_at":"2018-05-21T18:13:18Z","author_association":"OWNER","body":"Thanks for sharing the chatlogs. If you have used another neural network based chatbot or read a research paper yielding better conversational results, please send me the link to aid in my research on objective conversation quality metrics.\r\n\r\nIn response to your questions:\r\n\r\n1. The rnn_size for the v1 model was 1024, not 512. Check the hparams.json file in the trained_model_v1 folder. The values in the hparams.py file are always overwritten with the values from the json file. When training a new model, the json file is copied from the main seq2seq-chatbot folder to the specific model folder (to allow multiple models with different hparams to be used without changing any files).\r\n\r\nThe learning rate is higher because it is using regular SGD optimizer with a learning rate decay schedule (instead of Adam or RMSProp). This approach is similar to Google's [NMT model hparams](https://github.com/tensorflow/nmt/blob/master/nmt/standard_hparams/wmt16.json)\r\n\r\nBased on my observations I think that the best place to look to improve the model is by improving the word embeddings. I noticed that the embeddings trained on the cornell dataset are extremely densely clustered with the exception of a few very commonly used words (I used PCA projection with TensorBoard projector on the checkpoint file to observe this). This means that the model will receive very similar input vectors for a vast majority of the words in the vocabulary, which can degrade the ability to understand the questions. This is probably a symptom of not having enough word usage examples in the dataset.\r\n\r\nI think the best improvements will come from re-training using pre-trained word embeddings such as Word2Vec or GloVe. The new models I plan to upload will be trained using this approach (pre-trained embeddings), with the same layer count & size as before. I will also release models trained on other datasets.\r\n\r\n2. To train using a cluster or on parallel GPUs some additional changes to the model are necessary. I did not implement these for now. See [here](https://www.tensorflow.org/tutorials/seq2seq#multi-gpu_training).\r\n\r\nThanks!"}}
{"actor":{"display_login":"AbrahamSanders"}}