{"action":"created","issue":{"url":"https://api.github.com/repos/rwynn/monstache/issues/132","repository_url":"https://api.github.com/repos/rwynn/monstache","labels_url":"https://api.github.com/repos/rwynn/monstache/issues/132/labels{/name}","comments_url":"https://api.github.com/repos/rwynn/monstache/issues/132/comments","events_url":"https://api.github.com/repos/rwynn/monstache/issues/132/events","html_url":"https://github.com/rwynn/monstache/issues/132","id":381166959,"node_id":"MDU6SXNzdWUzODExNjY5NTk=","number":132,"title":"Partial update only","user":{"login":"thomasd3","id":14313218,"node_id":"MDQ6VXNlcjE0MzEzMjE4","avatar_url":"https://avatars3.githubusercontent.com/u/14313218?v=4","gravatar_id":"","url":"https://api.github.com/users/thomasd3","html_url":"https://github.com/thomasd3","followers_url":"https://api.github.com/users/thomasd3/followers","following_url":"https://api.github.com/users/thomasd3/following{/other_user}","gists_url":"https://api.github.com/users/thomasd3/gists{/gist_id}","starred_url":"https://api.github.com/users/thomasd3/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thomasd3/subscriptions","organizations_url":"https://api.github.com/users/thomasd3/orgs","repos_url":"https://api.github.com/users/thomasd3/repos","events_url":"https://api.github.com/users/thomasd3/events{/privacy}","received_events_url":"https://api.github.com/users/thomasd3/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":22,"created_at":"2018-11-15T13:44:13Z","updated_at":"2018-11-19T20:31:32Z","closed_at":null,"author_association":"NONE","body":"I have done another test: Mongo and ES with default values.\r\n\r\nI have inserted 79913 documents in Mongo\r\nES shows 15703\r\n\r\nMonstache's log doesn't show any error, it is full of lines like I posted below. It ran fine, then.. just stopped and never resumed.\r\n\r\nI just restarted the Monstache container, not touching anything else, and it is now continuing the process.\r\n\r\nWhat kind of event could make it believe it is up to date where a restart would behave differently?\r\n\r\n\r\n\r\n\r\n> 2018-11-15T04:33:15.813943344Z ,\r\n> 2018-11-15T04:33:15.863602133Z TRACE 2018/11/15 04:33:15 POST /_bulk HTTP/1.1\r\n> ,\r\n> 2018-11-15T04:33:15.863636783Z Host: elasticsearch:9200\r\n> ,\r\n> 2018-11-15T04:33:15.863648604Z User-Agent: elastic/6.2.11 (linux-amd64)\r\n> ,\r\n> 2018-11-15T04:33:15.863659574Z Content-Length: 353\r\n> ,\r\n> 2018-11-15T04:33:15.863670284Z Accept: application/json\r\n> ,\r\n> 2018-11-15T04:33:15.863681165Z Content-Type: application/x-ndjson\r\n> ,\r\n> 2018-11-15T04:33:15.863691950Z Accept-Encoding: gzip\r\n> ,\r\n> 2018-11-15T04:33:15.863702970Z \r\n> ,\r\n> 2018-11-15T04:33:15.813943344Z ,\r\n> 2018-11-15T04:33:15.863602133Z TRACE 2018/11/15 04:33:15 POST /_bulk HTTP/1.1\r\n> ,\r\n> 2018-11-15T04:33:15.863636783Z Host: elasticsearch:9200\r\n> ,\r\n> 2018-11-15T04:33:15.863648604Z User-Agent: elastic/6.2.11 (linux-amd64)\r\n> ,\r\n> 2018-11-15T04:33:15.863659574Z Content-Length: 353\r\n> ,\r\n> 2018-11-15T04:33:15.863670284Z Accept: application/json\r\n> ,\r\n> 2018-11-15T04:33:15.863681165Z Content-Type: application/x-ndjson\r\n> ,\r\n> 2018-11-15T04:33:15.863691950Z Accept-Encoding: gzip\r\n> ,\r\n> 2018-11-15T04:33:15.863702970Z \r\n> ,\r\n> 2018-11-15T04:33:15.863713770Z {\"index\":{\"_index\":\"test.pictures\",\"_id\":\"6t8Y22YJibT2581XdMqHHeEtQ7rB-sdAY1eZyWNSpJk\",\"_type\":\"pictureobject\",\"version\":6623940778571857932,\"version_type\":\"external\"}},\r\n> 2018-11-15T04:33:15.863729797Z {\"AlbumId\":\"ba1219cf-a4ac-48ef-ba44-97ea92acd215\",\"AlbumOffset\":1,\"CreatedOn\":\"2018-11-15T04:33:15.779Z\",\"Height\":1056,:1280},\r\n> 2018-11-15T04:33:15.863745391Z ,\r\n> 2018-11-15T04:33:15.975347512Z TRACE 2018/11/15 04:33:15 HTTP/1.1 200 OK\r\n> ,\r\n> 2018-11-15T04:33:15.975396401Z Content-Type: application/json; charset=UTF-8\r\n> ,\r\n> 2018-11-15T04:33:15.975409870Z \r\n> ,\r\n> 2018-11-15T04:33:15.975421687Z {\"took\":160,\"errors\":false,\"items\":[{\"index\":{\"_index\":\"test.pictures\",\"_type\":\"pictureobject\",\"_id\":\"UV5Va08voGCcigueWS33CAAOlVaPzccnGtQ16O748g0\",\"_version\":6623940778571857929,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":1,\"failed\":0},\"_seq_no\":2773,\"_primary_term\":1,\"status\":201}}]},\r\n> 2018-11-15T04:33:15.998852980Z TRACE 2018/11/15 04:33:15 HTTP/1.1 200 OK\r\n> ,\r\n> 2018-11-15T04:33:15.998884665Z Content-Type: application/json; charset=UTF-8\r\n> ,\r\n> 2018-11-15T04:33:15.998901749Z \r\n> ,\r\n> 2018-11-15T04:33:15.998916722Z {\"took\":134,\"errors\":false,\"items\":[{\"index\":{\"_index\":\"test.pictures\",\"_type\":\"pictureobject\",\"_id\":\"6t8Y22YJibT2581XdMqHHeEtQ7rB-sdAY1eZyWNSpJk\",\"_version\":6623940778571857932,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":1,\"failed\":0},\"_seq_no\":2886,\"_primary_term\":1,\"status\":201}}]},\r\n> 2018-11-15T04:33:16.011857085Z TRACE 2018/11/15 04:33:16 POST /_bulk HTTP/1.1\r\n> ,\r\n> 2018-11-15T04:33:16.011873086Z Host: elasticsearch:9200\r\n> ,\r\n> 2018-11-15T04:33:16.011877574Z User-Agent: elastic/6.2.11 (linux-amd64)\r\n> ,\r\n> 2018-11-15T04:33:16.011881519Z Content-Length: 353\r\n> ,\r\n> 2018-11-15T04:33:16.011885168Z Accept: application/json\r\n> ,\r\n> 2018-11-15T04:33:16.011892067Z Content-Type: application/x-ndjson\r\n> ,\r\n> 2018-11-15T04:33:16.011895881Z Accept-Encoding: gzip\r\n> ,\r\n> 2018-11-15T04:33:16.011899544Z \r\n> ,\r\n> 2018-11-15T04:33:16.011911735Z {\"index\":{\"_index\":\"test.pictures\",\"_id\":\"-6flUZrvARXzfwkpyZGuSZ3Jcjk+s7lIJUG86HucvaI\",\"_type\":\"pictureobject\",\"version\":6623940778571857943,\"version_type\":\"external\"}},\r\n> 2018-11-15T04:33:16.011917122Z {\"AlbumId\":\"bdabb811-e0a4-4683-be32-c5f9b0c1c550\",\"AlbumOffset\":1,\"CreatedOn\":\"2018-11-15T04:33:15.951Z\",\"Height\":1707,:1280},\r\n> 2018-11-15T04:33:16.011922212Z ,\r\n> 2018-11-15T04:33:16.168856954Z TRACE 2018/11/15 04:33:16 HTTP/1.1 200 OK\r\n> ,\r\n> 2018-11-15T04:33:16.168879239Z Content-Type: application/json; charset=UTF-8\r\n> ,\r\n> 2018-11-15T04:33:16.168886263Z \r\n> ,\r\n> 2018-11-15T04:33:16.168892682Z {\"took\":155,\"errors\":false,\"items\":[{\"index\":{\"_index\":\"test.pictures\",\"_type\":\"pictureobject\",\"_id\":\"-6flUZrvARXzfwkpyZGuSZ3Jcjk+s7lIJUG86HucvaI\",\"_version\":6623940778571857943,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":1,\"failed\":0},\"_seq_no\":2873,\"_primary_term\":1,\"status\":201}}]},\r\n> 2018-11-15T04:33:16.939082838Z TRACE 2018/11/15 04:33:16 POST /_bulk HTTP/1.1\r\n> ,\r\n> 2018-11-15T04:33:16.939114432Z Host: elasticsearch:9200\r\n> ,\r\n> 2018-11-15T04:33:16.939119081Z User-Agent: elastic/6.2.11 (linux-amd64)\r\n> ,\r\n> 2018-11-15T04:33:16.939123031Z Content-Length: 352\r\n> ,\r\n> 2018-11-15T04:33:16.939126620Z Accept: application/json\r\n> ,\r\n> 2018-11-15T04:33:16.939130441Z Content-Type: application/x-ndjson\r\n> ,\r\n> 2018-11-15T04:33:16.939133983Z Accept-Encoding: gzip\r\n> ,\r\n> 2018-11-15T04:33:16.939137506Z \r\n> ,\r\n> 2018-11-15T04:33:16.939141020Z {\"index\":{\"_index\":\"test.pictures\",\"_id\":\"io0YMW+63RHRth+lpSS9ZuFnCyEZBbqhs6RS7o9emoA\",\"_type\":\"pictureobject\",\"version\":6623940782866825221,\"version_type\":\"external\"}},\r\n> 2018-11-15T04:33:16.939159386Z {\"AlbumId\":\"bdabb811-e0a4-4683-be32-c5f9b0c1c550\",\"AlbumOffset\":2,\"CreatedOn\":\"2018-11-15T04:33:16.842Z\",\"Height\":936,:1280},\r\n> 2018-11-15T04:33:16.939166411Z ,\r\n> 2018-11-15T04:33:17.120868582Z TRACE 2018/11/15 04:33:17 HTTP/1.1 200 OK\r\n> ,\r\n> 2018-11-15T04:33:17.120909556Z Content-Type: application/json; charset=UTF-8\r\n> ,\r\n> 2018-11-15T04:33:17.120923217Z \r\n> ,\r\n> 2018-11-15T04:33:17.120934657Z {\"took\":180,\"errors\":false,\"items\":[{\"index\":{\"_index\":\"test.pictures\",\"_type\":\"pictureobject\",\"_id\":\"io0YMW+63RHRth+lpSS9ZuFnCyEZBbqhs6RS7o9emoA\",\"_version\":6623940782866825221,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":1,\"failed\":0},\"_seq_no\":2941,\"_primary_term\":1,\"status\":201}}]},\r\n> 2018-11-15T04:33:17.390380525Z TRACE 2018/11/15 04:33:17 POST /_bulk HTTP/1.1\r\n> ,\r\n> 2018-11-15T04:33:17.390418980Z Host: elasticsearch:9200\r\n> ,\r\n> 2018-11-15T04:33:17.390428162Z User-Agent: elastic/6.2.11 (linux-amd64)\r\n> ,\r\n> 2018-11-15T04:33:17.390435967Z Content-Length: 353\r\n> ,\r\n> 2018-11-15T04:33:17.390443256Z Accept: application/json\r\n> ,\r\n> 2018-11-15T04:33:17.390450596Z Content-Type: application/x-ndjson\r\n> ,\r\n> 2018-11-15T04:33:17.390457727Z Accept-Encoding: gzip\r\n> ,\r\n> 2018-11-15T04:33:17.390478861Z \r\n> ,\r\n> 2018-11-15T04:33:17.390486694Z {\"index\":{\"_index\":\"test.pictures\",\"_id\":\"rYE331Fy23wLP9PiwkYf86UZ1mQphIBC4wWdg-VqdsY\",\"_type\":\"pictureobject\",\"version\":6623940787161792515,\"version_type\":\"external\"}},\r\n> 2018-11-15T04:33:17.390496858Z {\"AlbumId\":\"b9a0522f-4a3e-4cbe-8c18-9a7259097ad5\",\"AlbumOffset\":9,\"CreatedOn\":\"2018-11-15T04:33:17.355Z\",\"Height\":1690,:1280},\r\n"},"comment":{"url":"https://api.github.com/repos/rwynn/monstache/issues/comments/440031768","html_url":"https://github.com/rwynn/monstache/issues/132#issuecomment-440031768","issue_url":"https://api.github.com/repos/rwynn/monstache/issues/132","id":440031768,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MDAzMTc2OA==","user":{"login":"rwynn","id":260672,"node_id":"MDQ6VXNlcjI2MDY3Mg==","avatar_url":"https://avatars2.githubusercontent.com/u/260672?v=4","gravatar_id":"","url":"https://api.github.com/users/rwynn","html_url":"https://github.com/rwynn","followers_url":"https://api.github.com/users/rwynn/followers","following_url":"https://api.github.com/users/rwynn/following{/other_user}","gists_url":"https://api.github.com/users/rwynn/gists{/gist_id}","starred_url":"https://api.github.com/users/rwynn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rwynn/subscriptions","organizations_url":"https://api.github.com/users/rwynn/orgs","repos_url":"https://api.github.com/users/rwynn/repos","events_url":"https://api.github.com/users/rwynn/events{/privacy}","received_events_url":"https://api.github.com/users/rwynn/received_events","type":"User","site_admin":false},"created_at":"2018-11-19T20:31:32Z","updated_at":"2018-11-19T20:31:32Z","author_association":"OWNER","body":"@thomasd3 I've been testing this and I think I may have been able to replicate the `stops` mid process.  I think it may have to do with the default timeout settings of `no timeout` and how this affects the mongodb driver.  Can you try the following settings and let me know if they help?\r\n\r\n```\r\n[mongo-dial-settings]\r\nread-timeout = 7\r\nwrite-timeout = 7\r\n\r\n[mongo-session-settings]\r\nsync-timeout = 7\r\n````\r\n\r\nThese are all 0 by default and basically turn off the timeouts in the driver.  But with the defaults I've noticed that the driver gets locked up after a while of usage.  With these timeouts set I see that this lock resolves itself and the processing continues.  \r\n"}}
{"actor":{"display_login":"rwynn"}}