{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/4563","repository_url":"https://api.github.com/repos/fchollet/keras","labels_url":"https://api.github.com/repos/fchollet/keras/issues/4563/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/4563/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/4563/events","html_url":"https://github.com/fchollet/keras/issues/4563","id":192792405,"number":4563,"title":"Very high / Nan loss when performing multiple sequence time series regression","user":{"login":"ankeshanand","id":2134563,"avatar_url":"https://avatars.githubusercontent.com/u/2134563?v=3","gravatar_id":"","url":"https://api.github.com/users/ankeshanand","html_url":"https://github.com/ankeshanand","followers_url":"https://api.github.com/users/ankeshanand/followers","following_url":"https://api.github.com/users/ankeshanand/following{/other_user}","gists_url":"https://api.github.com/users/ankeshanand/gists{/gist_id}","starred_url":"https://api.github.com/users/ankeshanand/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ankeshanand/subscriptions","organizations_url":"https://api.github.com/users/ankeshanand/orgs","repos_url":"https://api.github.com/users/ankeshanand/repos","events_url":"https://api.github.com/users/ankeshanand/events{/privacy}","received_events_url":"https://api.github.com/users/ankeshanand/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-12-01T09:22:06Z","updated_at":"2016-12-01T21:02:00Z","closed_at":null,"body":"I am training an LSTM model for multiple time-series regression. But, my losses are always either very high or Nan. I have tried several optimizers such as `rmsprop`, `adam` and `sgd`. Here's the script:\r\n\r\n````\r\nsgd = SGD(lr=0.0008, decay=1e-6, momentum=0.9, nesterov=True)\r\n# Build the Keras Model\r\nmodel = Sequential()\r\nmodel.add(LSTM(64, input_shape=(MAX_TIMESTEPS, MAX_FEATURES)))\r\nmodel.add(Dropout(0.3))\r\nmodel.add(Dense(1))\r\nmodel.compile(loss='mean_absolute_error',\r\n              optimizer=sgd)\r\n\r\nprint 'Training the LSTM model'\r\n\r\nbatch_size = 32\r\nmodel.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=20, validation_split=0.2)\r\nmae = model.evaluate(X_test, Y_test, batch_size=batch_size)\r\n````\r\n`X_train` is of the shape (N_SAMPLES_TRAIN, MAX_TIMESTEPS, MAX_FEATURES)\r\n`Y_train` is of the shape (N_SAMPLES)\r\n\r\nI should add that the Y values I am trying to predict are very high. Any idea where I might be going wroing?"},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/264293769","html_url":"https://github.com/fchollet/keras/issues/4563#issuecomment-264293769","issue_url":"https://api.github.com/repos/fchollet/keras/issues/4563","id":264293769,"user":{"login":"patyork","id":1304633,"avatar_url":"https://avatars.githubusercontent.com/u/1304633?v=3","gravatar_id":"","url":"https://api.github.com/users/patyork","html_url":"https://github.com/patyork","followers_url":"https://api.github.com/users/patyork/followers","following_url":"https://api.github.com/users/patyork/following{/other_user}","gists_url":"https://api.github.com/users/patyork/gists{/gist_id}","starred_url":"https://api.github.com/users/patyork/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/patyork/subscriptions","organizations_url":"https://api.github.com/users/patyork/orgs","repos_url":"https://api.github.com/users/patyork/repos","events_url":"https://api.github.com/users/patyork/events{/privacy}","received_events_url":"https://api.github.com/users/patyork/received_events","type":"User","site_admin":false},"created_at":"2016-12-01T21:01:59Z","updated_at":"2016-12-01T21:01:59Z","body":"When working with very long sequences (e.g. a large MAX_TIMESTEPS), it's pretty common for underflow/overflow issues during the early stages of training, where the error is large and gets larger (or inversely, smaller and gets smaller) with each timestep. This causes NaNs, INFs, and the like. This is due to random state of the model at the start of the training, and the loss function accruing for longer with longer sequences.\r\n\r\nThis can be mitigated by:\r\n- Training on the shortest sequences first, and adding longer sequences to the training as the model moves from random towards a local minimum\r\n- Initializing the weights intelligently such that the model starts out in a less-random state\r\n- Using a clipped (min()'d or max()'d) loss function that keeps the loss within the realm of float numbers"}}
{"actor":{"display_login":"patyork"}}