{"action":"created","issue":{"url":"https://api.github.com/repos/spotify/scio/issues/1262","repository_url":"https://api.github.com/repos/spotify/scio","labels_url":"https://api.github.com/repos/spotify/scio/issues/1262/labels{/name}","comments_url":"https://api.github.com/repos/spotify/scio/issues/1262/comments","events_url":"https://api.github.com/repos/spotify/scio/issues/1262/events","html_url":"https://github.com/spotify/scio/issues/1262","id":344341711,"node_id":"MDU6SXNzdWUzNDQzNDE3MTE=","number":1262,"title":"AsyncDoFn deadlock","user":{"login":"welvet","id":1097654,"node_id":"MDQ6VXNlcjEwOTc2NTQ=","avatar_url":"https://avatars0.githubusercontent.com/u/1097654?v=4","gravatar_id":"","url":"https://api.github.com/users/welvet","html_url":"https://github.com/welvet","followers_url":"https://api.github.com/users/welvet/followers","following_url":"https://api.github.com/users/welvet/following{/other_user}","gists_url":"https://api.github.com/users/welvet/gists{/gist_id}","starred_url":"https://api.github.com/users/welvet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/welvet/subscriptions","organizations_url":"https://api.github.com/users/welvet/orgs","repos_url":"https://api.github.com/users/welvet/repos","events_url":"https://api.github.com/users/welvet/events{/privacy}","received_events_url":"https://api.github.com/users/welvet/received_events","type":"User","site_admin":false},"labels":[{"id":996733407,"node_id":"MDU6TGFiZWw5OTY3MzM0MDc=","url":"https://api.github.com/repos/spotify/scio/labels/P1","name":"P1","color":"d93f0b","default":false},{"id":192446851,"node_id":"MDU6TGFiZWwxOTI0NDY4NTE=","url":"https://api.github.com/repos/spotify/scio/labels/bug","name":"bug","color":"d73a4a","default":true},{"id":192446855,"node_id":"MDU6TGFiZWwxOTI0NDY4NTU=","url":"https://api.github.com/repos/spotify/scio/labels/invalid","name":"invalid","color":"e4e669","default":true}],"state":"closed","locked":false,"assignee":{"login":"nevillelyh","id":468514,"node_id":"MDQ6VXNlcjQ2ODUxNA==","avatar_url":"https://avatars0.githubusercontent.com/u/468514?v=4","gravatar_id":"","url":"https://api.github.com/users/nevillelyh","html_url":"https://github.com/nevillelyh","followers_url":"https://api.github.com/users/nevillelyh/followers","following_url":"https://api.github.com/users/nevillelyh/following{/other_user}","gists_url":"https://api.github.com/users/nevillelyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nevillelyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nevillelyh/subscriptions","organizations_url":"https://api.github.com/users/nevillelyh/orgs","repos_url":"https://api.github.com/users/nevillelyh/repos","events_url":"https://api.github.com/users/nevillelyh/events{/privacy}","received_events_url":"https://api.github.com/users/nevillelyh/received_events","type":"User","site_admin":false},"assignees":[{"login":"nevillelyh","id":468514,"node_id":"MDQ6VXNlcjQ2ODUxNA==","avatar_url":"https://avatars0.githubusercontent.com/u/468514?v=4","gravatar_id":"","url":"https://api.github.com/users/nevillelyh","html_url":"https://github.com/nevillelyh","followers_url":"https://api.github.com/users/nevillelyh/followers","following_url":"https://api.github.com/users/nevillelyh/following{/other_user}","gists_url":"https://api.github.com/users/nevillelyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nevillelyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nevillelyh/subscriptions","organizations_url":"https://api.github.com/users/nevillelyh/orgs","repos_url":"https://api.github.com/users/nevillelyh/repos","events_url":"https://api.github.com/users/nevillelyh/events{/privacy}","received_events_url":"https://api.github.com/users/nevillelyh/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/spotify/scio/milestones/7","html_url":"https://github.com/spotify/scio/milestone/7","labels_url":"https://api.github.com/repos/spotify/scio/milestones/7/labels","id":3539467,"node_id":"MDk6TWlsZXN0b25lMzUzOTQ2Nw==","number":7,"title":"0.5.7","description":"","creator":{"login":"regadas","id":163899,"node_id":"MDQ6VXNlcjE2Mzg5OQ==","avatar_url":"https://avatars2.githubusercontent.com/u/163899?v=4","gravatar_id":"","url":"https://api.github.com/users/regadas","html_url":"https://github.com/regadas","followers_url":"https://api.github.com/users/regadas/followers","following_url":"https://api.github.com/users/regadas/following{/other_user}","gists_url":"https://api.github.com/users/regadas/gists{/gist_id}","starred_url":"https://api.github.com/users/regadas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/regadas/subscriptions","organizations_url":"https://api.github.com/users/regadas/orgs","repos_url":"https://api.github.com/users/regadas/repos","events_url":"https://api.github.com/users/regadas/events{/privacy}","received_events_url":"https://api.github.com/users/regadas/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":10,"state":"open","created_at":"2018-08-01T14:39:14Z","updated_at":"2018-08-15T13:19:46Z","due_on":"2018-08-15T07:00:00Z","closed_at":null},"comments":3,"created_at":"2018-07-25T08:24:23Z","updated_at":"2018-08-15T13:19:46Z","closed_at":"2018-08-15T13:19:46Z","author_association":"NONE","body":"We're using Dataflow to build our pipelines. Recently we had an incident with one of them.\r\nOur Dataflow job stopped consuming messages and had 0 CPU load. \r\nWe asked Google support to assist us and they pointed to a deadlock in `ScalaAsyncDoFn.waitForFutures` method call.\r\nEven that `waitForFutures` deadlock was likely a result of our poorly written `processElement` there are still some concerns about `ScalaAsyncDoFn`.\r\nGoogle pointed us that `DoFn`'s should be stateless and we should avoid adding threading, blocking, queues or pooling.\r\nAnd I'm quoting the most interesting part of message in Google Case:\r\n\r\nThe DoFn should not depend on the state of others elements within the bundle. If the process of one element fails, dataflow may rebundle the elements into a different set, and the DoFn instance may be used on another set. The elements may be re-distributed to other bundles. Per the \"No Global Shared State\" section of the ParDo API documentation: \"programs should not access any mutable static variable state in their DoFn, without understanding that the Java processes for the main program and workers will each have its own independent copy of such state\"   \r\n   \r\nThe normal recommended pattern of DoFn is as you suspected: \"process messages synchronously during processElement invocation\".  There can be exceptions, but I am not familiar with them or the rest of your application enough to recommend them at this time.  Here are some more guidelines from our team that I hope are helpful. \r\n\r\n1) There are no guarantees of how many times processElement() will be called between startBundle() and finishBundle(). There is no guarantee the number will stay the same if a bundle is re-tried.\r\n2) Avoid adding threading, blocking, queues or pooling. Let Dataflow handle concurrency\r\n3) Make sure processElement is thread-safe\r\n\r\nI have links to internal incident ticket and Google Case but will not post them here."},"comment":{"url":"https://api.github.com/repos/spotify/scio/issues/comments/413194906","html_url":"https://github.com/spotify/scio/issues/1262#issuecomment-413194906","issue_url":"https://api.github.com/repos/spotify/scio/issues/1262","id":413194906,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMzE5NDkwNg==","user":{"login":"nevillelyh","id":468514,"node_id":"MDQ6VXNlcjQ2ODUxNA==","avatar_url":"https://avatars0.githubusercontent.com/u/468514?v=4","gravatar_id":"","url":"https://api.github.com/users/nevillelyh","html_url":"https://github.com/nevillelyh","followers_url":"https://api.github.com/users/nevillelyh/followers","following_url":"https://api.github.com/users/nevillelyh/following{/other_user}","gists_url":"https://api.github.com/users/nevillelyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nevillelyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nevillelyh/subscriptions","organizations_url":"https://api.github.com/users/nevillelyh/orgs","repos_url":"https://api.github.com/users/nevillelyh/repos","events_url":"https://api.github.com/users/nevillelyh/events{/privacy}","received_events_url":"https://api.github.com/users/nevillelyh/received_events","type":"User","site_admin":false},"created_at":"2018-08-15T13:19:46Z","updated_at":"2018-08-15T13:19:46Z","author_association":"MEMBER","body":"Closing for now since it might be misuse of the API and the user has worked around it."}}
{"actor":{"display_login":"nevillelyh"}}