{"action":"opened","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/2068","repository_url":"https://api.github.com/repos/fchollet/keras","labels_url":"https://api.github.com/repos/fchollet/keras/issues/2068/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/2068/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/2068/events","html_url":"https://github.com/fchollet/keras/issues/2068","id":143282729,"number":2068,"title":"Graph vs Sequential Loss","user":{"login":"bkj","id":6086781,"avatar_url":"https://avatars.githubusercontent.com/u/6086781?v=3","gravatar_id":"","url":"https://api.github.com/users/bkj","html_url":"https://github.com/bkj","followers_url":"https://api.github.com/users/bkj/followers","following_url":"https://api.github.com/users/bkj/following{/other_user}","gists_url":"https://api.github.com/users/bkj/gists{/gist_id}","starred_url":"https://api.github.com/users/bkj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bkj/subscriptions","organizations_url":"https://api.github.com/users/bkj/orgs","repos_url":"https://api.github.com/users/bkj/repos","events_url":"https://api.github.com/users/bkj/events{/privacy}","received_events_url":"https://api.github.com/users/bkj/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-03-24T15:46:58Z","updated_at":"2016-03-24T15:46:58Z","closed_at":null,"body":"Hi all,\r\n\r\nI'm trying to figure out the precise differences between `Sequential` and `Graph` models, and I'm a little bit confused about a particular behavior that I'm encountering.   First I implement a `Sequential` model:\r\n\r\n```\r\n        model = Sequential()\r\n        model.add(\r\n            Embedding(\r\n                trainable  = False,\r\n                input_dim  = 100000 \r\n                output_dim = 500 \r\n                mask_zero  = True,\r\n            )\r\n        )\r\n        \r\n        model.add(TimeDistributedMerge(mode='ave'))\r\n        \r\n        model.add(Dense(200, activation='relu'))\r\n        model.add(Dropout(0.5))\r\n        model.add(Dense(200, activation='relu'))\r\n        model.add(Dropout(0.5))\r\n        \r\n        model.add(Dense(5, activation='softmax'))\r\n        model.compile(loss='categorical_crossentropy', optimizer='adam')\r\n```\r\n\r\nI train this for a single epoch and get:\r\n\r\n```\r\nfitist = model.fit(\r\n    sX_train, sy_train,\r\n    nb_epoch        = 1,\r\n    batch_size      = 256\r\n)\r\n\r\n171048/171048 [==============================] - 18s - loss: 1.5751\r\n```\r\n\r\nNow I re-implement the same model as a Graph, where all nodes are shared w/ the input and output replicated twice.  I weight the loss at `0.5`, and I would expect the results to be (roughly) the same.\r\n\r\n```\r\n       model = Graph()\r\n        \r\n        model.add_input(name='labeled_1', input_shape=(embedding['input_dim'],), dtype=int)\r\n        model.add_input(name='labeled_2', input_shape=(embedding['input_dim'],), dtype=int)\r\n        \r\n        model.add_shared_node(\r\n            Embedding(\r\n                trainable  = False,\r\n                input_dim  = 100000 \r\n                output_dim = 500, \r\n                mask_zero  = True\r\n            ),\r\n            name='emb',\r\n            inputs=['labeled_1', 'labeled_2'], outputs=['emb_1', 'emb_2']\r\n        )\r\n        \r\n        model.add_shared_node(TimeDistributedMerge(mode='ave'),\r\n            name='tdm', inputs=['emb_1', 'emb_2'], outputs=['tdm_1', 'tdm_2'])\r\n        \r\n        model.add_shared_node(Dense(200, activation='relu'), \r\n            name='dense1', inputs=['tdm_1', 'tdm_2'], outputs=['dense1_1', 'dense1_2'])\r\n        model.add_shared_node(Dropout(0.5),\r\n            name='do1', inputs=['dense1_1', 'dense1_2'], outputs=['do1_1', 'do1_2'])\r\n        \r\n        model.add_shared_node(Dense(200, activation='relu'), \r\n            name='dense2', inputs=['do1_1', 'do1_2'], outputs=['dense2_1', 'dense2_2'])\r\n        model.add_shared_node(Dropout(0.5),\r\n            name='do2', inputs=['dense2_1', 'dense2_2'], outputs=['do2_1', 'do2_2'])\r\n        model.add_shared_node(Dense(5, activation='softmax'), name='densef', inputs=['do2_1', 'do2_2'], outputs=['densef_1', 'densef_2'])\r\n        model.add_output(name=\"lab_1\", input=\"densef_1\")\r\n        model.add_output(name=\"lab_2\", input=\"densef_2\")\r\n        \r\n        model.compile(loss={\r\n            \"lab_1\" : 'categorical_crossentropy',\r\n            \"lab_2\" : 'categorical_crossentropy'\r\n        }, loss_weights={\"lab_1\" : 0.5, \"lab_2\" : 0.5}, optimizer='adam')\r\n```\r\n\r\nTrain this model:\r\n\r\n```\r\nmodel.fit({\r\n    \"labeled_1\" : sX_train,\r\n    \"labeled_2\" : sX_train,\r\n    \"lab_1\"     : sy_train,\r\n    \"lab_2\"     : sy_train,\r\n}, nb_epoch=1, batch_size=256)\r\n\r\n171048/171048 [==============================] - 139s - loss: 1.2544\r\n```\r\n\r\nSo this second model takes 10x longer, has lower training loss but also significantly lower out of sample accuracy.   Does anyone have insight into what might be going on?\r\n"}}