{"action":"created","comment":{"url":"https://api.github.com/repos/tensorflow/swift-models/pulls/comments/188669429","pull_request_review_id":120694842,"id":188669429,"diff_hunk":"@@ -76,20 +82,32 @@ func main() {\n \n         // Backward pass.\n         let dz2 = predictions - labels\n-        let dw2 = h1.transposed(withPermutations: 1, 0) ⊗ dz2\n-        let db2 = dz2.sum(squeezingAxes: 0)\n+        let dw2 = h1.transposed(withPermutations: 1, 0) ⊗ dz2  / batchSize\n+        let db2 = dz2.sum(squeezingAxes: 0) / batchSize\n         let dz1 = dz2.dot(w2.transposed(withPermutations: 1, 0)) * h1 * (1 - h1)\n-        let dw1 = images.transposed(withPermutations: 1, 0) ⊗ dz1\n-        let db1 = dz1.sum(squeezingAxes: 0)\n+        let dw1 = images.transposed(withPermutations: 1, 0) ⊗ dz1  / batchSize\n+        let db1 = dz1.sum(squeezingAxes: 0)  / batchSize\n \n         // Gradient descent.\n         w1 -= dw1 * learningRate\n         b1 -= db1 * learningRate\n         w2 -= dw2 * learningRate\n         b2 -= db2 * learningRate\n \n-        // Update loss.\n-        loss = dz2.squared().mean(squeezingAxes: 1, 0).scalarized()\n+        // Update the cross-entropy loss.\n+        // Let m be the batch size, y be the target labels and A be the\n+        // predictions.  The formula expressed in TF expression is:\n+        // 1/m * tf.reduce_sum(- y * tf.log(A) - (1-y) * tf.log(1-A))\n+        let part1 = -(labels) * log(predictions)\n+        let part2 = -(1 - labels) * log(1 - predictions)\n+        loss = ((part1 + part2).sum(squeezingAxes: 1, 0) / batchSize)\n+          .scalarized()\n+        // To print out the loss value per iteration, uncomment the following\n+        // code.\n+        // FIXME: Fix runtime hanging when we print loss directly instead of\n+        // printing via lossTensor.\n+        // let lossTensor = Tensor<Float>(loss)\n+        // print(lossTensor)","path":"MNIST/MNIST.swift","position":73,"original_position":73,"commit_id":"e9a12684f38389aeaa03631f0515379be9a73fa3","original_commit_id":"e9a12684f38389aeaa03631f0515379be9a73fa3","user":{"login":"dan-zheng","id":5590046,"avatar_url":"https://avatars0.githubusercontent.com/u/5590046?v=4","gravatar_id":"","url":"https://api.github.com/users/dan-zheng","html_url":"https://github.com/dan-zheng","followers_url":"https://api.github.com/users/dan-zheng/followers","following_url":"https://api.github.com/users/dan-zheng/following{/other_user}","gists_url":"https://api.github.com/users/dan-zheng/gists{/gist_id}","starred_url":"https://api.github.com/users/dan-zheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dan-zheng/subscriptions","organizations_url":"https://api.github.com/users/dan-zheng/orgs","repos_url":"https://api.github.com/users/dan-zheng/repos","events_url":"https://api.github.com/users/dan-zheng/events{/privacy}","received_events_url":"https://api.github.com/users/dan-zheng/received_events","type":"User","site_admin":false},"body":"Uncommenting these two lines to `lossTensor` causes runtime hanging for me.\r\nIs this not the case for you?","created_at":"2018-05-16T15:28:16Z","updated_at":"2018-05-16T15:28:17Z","html_url":"https://github.com/tensorflow/swift-models/pull/6#discussion_r188669429","pull_request_url":"https://api.github.com/repos/tensorflow/swift-models/pulls/6","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/tensorflow/swift-models/pulls/comments/188669429"},"html":{"href":"https://github.com/tensorflow/swift-models/pull/6#discussion_r188669429"},"pull_request":{"href":"https://api.github.com/repos/tensorflow/swift-models/pulls/6"}}},"pull_request":{"url":"https://api.github.com/repos/tensorflow/swift-models/pulls/6","id":188462014,"html_url":"https://github.com/tensorflow/swift-models/pull/6","diff_url":"https://github.com/tensorflow/swift-models/pull/6.diff","patch_url":"https://github.com/tensorflow/swift-models/pull/6.patch","issue_url":"https://api.github.com/repos/tensorflow/swift-models/issues/6","number":6,"state":"open","locked":false,"title":"A few fixes to the MNIST model:","user":{"login":"mhong","id":969799,"avatar_url":"https://avatars3.githubusercontent.com/u/969799?v=4","gravatar_id":"","url":"https://api.github.com/users/mhong","html_url":"https://github.com/mhong","followers_url":"https://api.github.com/users/mhong/followers","following_url":"https://api.github.com/users/mhong/following{/other_user}","gists_url":"https://api.github.com/users/mhong/gists{/gist_id}","starred_url":"https://api.github.com/users/mhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mhong/subscriptions","organizations_url":"https://api.github.com/users/mhong/orgs","repos_url":"https://api.github.com/users/mhong/repos","events_url":"https://api.github.com/users/mhong/events{/privacy}","received_events_url":"https://api.github.com/users/mhong/received_events","type":"User","site_admin":false},"body":"1. Normalized a few gradient values by the bach size.\r\n2. Changed the loss function from MSE to cross entropy, which works with the\r\nhard-coded gradients.\r\n\r\nThe change of loss values across training iterations before this CL:\r\n\r\nBegin training for 20 iterations.\r\n0.9\r\n0.1\r\n0.1\r\n0.1\r\n0.1\r\n0.1\r\n0.1\r\n0.1\r\n0.1\r\n0.177527\r\n0.25868\r\n0.581863\r\n0.18193\r\n0.1\r\n0.1\r\n0.1\r\n0.1\r\n0.1\r\n0.177527\r\n0.1\r\n0.1\r\n\r\nAfter this CL:\r\n\r\nBegin training for 20 iterations.\r\nnan\r\n113.156\r\n88.0401\r\n62.9524\r\n38.1512\r\n16.3276\r\n5.25584\r\n3.61207\r\n3.37472\r\n3.30145\r\n3.2731\r\n3.26038\r\n3.2549\r\n3.25211\r\n3.25062\r\n3.2499\r\n3.24947\r\n3.2495\r\n3.24934\r\n3.24926\r\n3.24926\r\n\r\nThere might be further tuning opportunities with learning rate, etc, but is beyond the scope of this CL.\r\n\r\nWorking on this CL exposed a few compiler bugs. Left FIXME comments accordingly,\r\nand will file bugs.","created_at":"2018-05-16T14:56:50Z","updated_at":"2018-05-16T15:28:17Z","closed_at":null,"merged_at":null,"merge_commit_sha":"aca40ce87d6c4e30fd3a155bb5f388b24c3f4dc6","assignee":null,"assignees":[],"requested_reviewers":[{"login":"rxwei","id":856507,"avatar_url":"https://avatars0.githubusercontent.com/u/856507?v=4","gravatar_id":"","url":"https://api.github.com/users/rxwei","html_url":"https://github.com/rxwei","followers_url":"https://api.github.com/users/rxwei/followers","following_url":"https://api.github.com/users/rxwei/following{/other_user}","gists_url":"https://api.github.com/users/rxwei/gists{/gist_id}","starred_url":"https://api.github.com/users/rxwei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rxwei/subscriptions","organizations_url":"https://api.github.com/users/rxwei/orgs","repos_url":"https://api.github.com/users/rxwei/repos","events_url":"https://api.github.com/users/rxwei/events{/privacy}","received_events_url":"https://api.github.com/users/rxwei/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/tensorflow/swift-models/pulls/6/commits","review_comments_url":"https://api.github.com/repos/tensorflow/swift-models/pulls/6/comments","review_comment_url":"https://api.github.com/repos/tensorflow/swift-models/pulls/comments{/number}","comments_url":"https://api.github.com/repos/tensorflow/swift-models/issues/6/comments","statuses_url":"https://api.github.com/repos/tensorflow/swift-models/statuses/e9a12684f38389aeaa03631f0515379be9a73fa3","head":{"label":"tensorflow:mnist_model","ref":"mnist_model","sha":"e9a12684f38389aeaa03631f0515379be9a73fa3","user":{"login":"tensorflow","id":15658638,"avatar_url":"https://avatars1.githubusercontent.com/u/15658638?v=4","gravatar_id":"","url":"https://api.github.com/users/tensorflow","html_url":"https://github.com/tensorflow","followers_url":"https://api.github.com/users/tensorflow/followers","following_url":"https://api.github.com/users/tensorflow/following{/other_user}","gists_url":"https://api.github.com/users/tensorflow/gists{/gist_id}","starred_url":"https://api.github.com/users/tensorflow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tensorflow/subscriptions","organizations_url":"https://api.github.com/users/tensorflow/orgs","repos_url":"https://api.github.com/users/tensorflow/repos","events_url":"https://api.github.com/users/tensorflow/events{/privacy}","received_events_url":"https://api.github.com/users/tensorflow/received_events","type":"Organization","site_admin":false},"repo":{"id":130903312,"name":"swift-models","full_name":"tensorflow/swift-models","owner":{"login":"tensorflow","id":15658638,"avatar_url":"https://avatars1.githubusercontent.com/u/15658638?v=4","gravatar_id":"","url":"https://api.github.com/users/tensorflow","html_url":"https://github.com/tensorflow","followers_url":"https://api.github.com/users/tensorflow/followers","following_url":"https://api.github.com/users/tensorflow/following{/other_user}","gists_url":"https://api.github.com/users/tensorflow/gists{/gist_id}","starred_url":"https://api.github.com/users/tensorflow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tensorflow/subscriptions","organizations_url":"https://api.github.com/users/tensorflow/orgs","repos_url":"https://api.github.com/users/tensorflow/repos","events_url":"https://api.github.com/users/tensorflow/events{/privacy}","received_events_url":"https://api.github.com/users/tensorflow/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/tensorflow/swift-models","description":"Example models built using Swift for TensorFlow","fork":false,"url":"https://api.github.com/repos/tensorflow/swift-models","forks_url":"https://api.github.com/repos/tensorflow/swift-models/forks","keys_url":"https://api.github.com/repos/tensorflow/swift-models/keys{/key_id}","collaborators_url":"https://api.github.com/repos/tensorflow/swift-models/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/tensorflow/swift-models/teams","hooks_url":"https://api.github.com/repos/tensorflow/swift-models/hooks","issue_events_url":"https://api.github.com/repos/tensorflow/swift-models/issues/events{/number}","events_url":"https://api.github.com/repos/tensorflow/swift-models/events","assignees_url":"https://api.github.com/repos/tensorflow/swift-models/assignees{/user}","branches_url":"https://api.github.com/repos/tensorflow/swift-models/branches{/branch}","tags_url":"https://api.github.com/repos/tensorflow/swift-models/tags","blobs_url":"https://api.github.com/repos/tensorflow/swift-models/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/tensorflow/swift-models/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/tensorflow/swift-models/git/refs{/sha}","trees_url":"https://api.github.com/repos/tensorflow/swift-models/git/trees{/sha}","statuses_url":"https://api.github.com/repos/tensorflow/swift-models/statuses/{sha}","languages_url":"https://api.github.com/repos/tensorflow/swift-models/languages","stargazers_url":"https://api.github.com/repos/tensorflow/swift-models/stargazers","contributors_url":"https://api.github.com/repos/tensorflow/swift-models/contributors","subscribers_url":"https://api.github.com/repos/tensorflow/swift-models/subscribers","subscription_url":"https://api.github.com/repos/tensorflow/swift-models/subscription","commits_url":"https://api.github.com/repos/tensorflow/swift-models/commits{/sha}","git_commits_url":"https://api.github.com/repos/tensorflow/swift-models/git/commits{/sha}","comments_url":"https://api.github.com/repos/tensorflow/swift-models/comments{/number}","issue_comment_url":"https://api.github.com/repos/tensorflow/swift-models/issues/comments{/number}","contents_url":"https://api.github.com/repos/tensorflow/swift-models/contents/{+path}","compare_url":"https://api.github.com/repos/tensorflow/swift-models/compare/{base}...{head}","merges_url":"https://api.github.com/repos/tensorflow/swift-models/merges","archive_url":"https://api.github.com/repos/tensorflow/swift-models/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/tensorflow/swift-models/downloads","issues_url":"https://api.github.com/repos/tensorflow/swift-models/issues{/number}","pulls_url":"https://api.github.com/repos/tensorflow/swift-models/pulls{/number}","milestones_url":"https://api.github.com/repos/tensorflow/swift-models/milestones{/number}","notifications_url":"https://api.github.com/repos/tensorflow/swift-models/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/tensorflow/swift-models/labels{/name}","releases_url":"https://api.github.com/repos/tensorflow/swift-models/releases{/id}","deployments_url":"https://api.github.com/repos/tensorflow/swift-models/deployments","created_at":"2018-04-24T19:21:41Z","updated_at":"2018-05-16T14:15:52Z","pushed_at":"2018-05-16T14:56:51Z","git_url":"git://github.com/tensorflow/swift-models.git","ssh_url":"git@github.com:tensorflow/swift-models.git","clone_url":"https://github.com/tensorflow/swift-models.git","svn_url":"https://github.com/tensorflow/swift-models","homepage":"","size":9668,"stargazers_count":98,"watchers_count":98,"language":"Swift","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":10,"mirror_url":null,"archived":false,"open_issues_count":3,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":10,"open_issues":3,"watchers":98,"default_branch":"master"}},"base":{"label":"tensorflow:master","ref":"master","sha":"fc7a179ad98028189928782527bc886504b20399","user":{"login":"tensorflow","id":15658638,"avatar_url":"https://avatars1.githubusercontent.com/u/15658638?v=4","gravatar_id":"","url":"https://api.github.com/users/tensorflow","html_url":"https://github.com/tensorflow","followers_url":"https://api.github.com/users/tensorflow/followers","following_url":"https://api.github.com/users/tensorflow/following{/other_user}","gists_url":"https://api.github.com/users/tensorflow/gists{/gist_id}","starred_url":"https://api.github.com/users/tensorflow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tensorflow/subscriptions","organizations_url":"https://api.github.com/users/tensorflow/orgs","repos_url":"https://api.github.com/users/tensorflow/repos","events_url":"https://api.github.com/users/tensorflow/events{/privacy}","received_events_url":"https://api.github.com/users/tensorflow/received_events","type":"Organization","site_admin":false},"repo":{"id":130903312,"name":"swift-models","full_name":"tensorflow/swift-models","owner":{"login":"tensorflow","id":15658638,"avatar_url":"https://avatars1.githubusercontent.com/u/15658638?v=4","gravatar_id":"","url":"https://api.github.com/users/tensorflow","html_url":"https://github.com/tensorflow","followers_url":"https://api.github.com/users/tensorflow/followers","following_url":"https://api.github.com/users/tensorflow/following{/other_user}","gists_url":"https://api.github.com/users/tensorflow/gists{/gist_id}","starred_url":"https://api.github.com/users/tensorflow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tensorflow/subscriptions","organizations_url":"https://api.github.com/users/tensorflow/orgs","repos_url":"https://api.github.com/users/tensorflow/repos","events_url":"https://api.github.com/users/tensorflow/events{/privacy}","received_events_url":"https://api.github.com/users/tensorflow/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/tensorflow/swift-models","description":"Example models built using Swift for TensorFlow","fork":false,"url":"https://api.github.com/repos/tensorflow/swift-models","forks_url":"https://api.github.com/repos/tensorflow/swift-models/forks","keys_url":"https://api.github.com/repos/tensorflow/swift-models/keys{/key_id}","collaborators_url":"https://api.github.com/repos/tensorflow/swift-models/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/tensorflow/swift-models/teams","hooks_url":"https://api.github.com/repos/tensorflow/swift-models/hooks","issue_events_url":"https://api.github.com/repos/tensorflow/swift-models/issues/events{/number}","events_url":"https://api.github.com/repos/tensorflow/swift-models/events","assignees_url":"https://api.github.com/repos/tensorflow/swift-models/assignees{/user}","branches_url":"https://api.github.com/repos/tensorflow/swift-models/branches{/branch}","tags_url":"https://api.github.com/repos/tensorflow/swift-models/tags","blobs_url":"https://api.github.com/repos/tensorflow/swift-models/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/tensorflow/swift-models/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/tensorflow/swift-models/git/refs{/sha}","trees_url":"https://api.github.com/repos/tensorflow/swift-models/git/trees{/sha}","statuses_url":"https://api.github.com/repos/tensorflow/swift-models/statuses/{sha}","languages_url":"https://api.github.com/repos/tensorflow/swift-models/languages","stargazers_url":"https://api.github.com/repos/tensorflow/swift-models/stargazers","contributors_url":"https://api.github.com/repos/tensorflow/swift-models/contributors","subscribers_url":"https://api.github.com/repos/tensorflow/swift-models/subscribers","subscription_url":"https://api.github.com/repos/tensorflow/swift-models/subscription","commits_url":"https://api.github.com/repos/tensorflow/swift-models/commits{/sha}","git_commits_url":"https://api.github.com/repos/tensorflow/swift-models/git/commits{/sha}","comments_url":"https://api.github.com/repos/tensorflow/swift-models/comments{/number}","issue_comment_url":"https://api.github.com/repos/tensorflow/swift-models/issues/comments{/number}","contents_url":"https://api.github.com/repos/tensorflow/swift-models/contents/{+path}","compare_url":"https://api.github.com/repos/tensorflow/swift-models/compare/{base}...{head}","merges_url":"https://api.github.com/repos/tensorflow/swift-models/merges","archive_url":"https://api.github.com/repos/tensorflow/swift-models/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/tensorflow/swift-models/downloads","issues_url":"https://api.github.com/repos/tensorflow/swift-models/issues{/number}","pulls_url":"https://api.github.com/repos/tensorflow/swift-models/pulls{/number}","milestones_url":"https://api.github.com/repos/tensorflow/swift-models/milestones{/number}","notifications_url":"https://api.github.com/repos/tensorflow/swift-models/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/tensorflow/swift-models/labels{/name}","releases_url":"https://api.github.com/repos/tensorflow/swift-models/releases{/id}","deployments_url":"https://api.github.com/repos/tensorflow/swift-models/deployments","created_at":"2018-04-24T19:21:41Z","updated_at":"2018-05-16T14:15:52Z","pushed_at":"2018-05-16T14:56:51Z","git_url":"git://github.com/tensorflow/swift-models.git","ssh_url":"git@github.com:tensorflow/swift-models.git","clone_url":"https://github.com/tensorflow/swift-models.git","svn_url":"https://github.com/tensorflow/swift-models","homepage":"","size":9668,"stargazers_count":98,"watchers_count":98,"language":"Swift","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":10,"mirror_url":null,"archived":false,"open_issues_count":3,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":10,"open_issues":3,"watchers":98,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/tensorflow/swift-models/pulls/6"},"html":{"href":"https://github.com/tensorflow/swift-models/pull/6"},"issue":{"href":"https://api.github.com/repos/tensorflow/swift-models/issues/6"},"comments":{"href":"https://api.github.com/repos/tensorflow/swift-models/issues/6/comments"},"review_comments":{"href":"https://api.github.com/repos/tensorflow/swift-models/pulls/6/comments"},"review_comment":{"href":"https://api.github.com/repos/tensorflow/swift-models/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/tensorflow/swift-models/pulls/6/commits"},"statuses":{"href":"https://api.github.com/repos/tensorflow/swift-models/statuses/e9a12684f38389aeaa03631f0515379be9a73fa3"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"dan-zheng"}}