{"action":"opened","issue":{"url":"https://api.github.com/repos/minio/minio/issues/3232","repository_url":"https://api.github.com/repos/minio/minio","labels_url":"https://api.github.com/repos/minio/minio/issues/3232/labels{/name}","comments_url":"https://api.github.com/repos/minio/minio/issues/3232/comments","events_url":"https://api.github.com/repos/minio/minio/issues/3232/events","html_url":"https://github.com/minio/minio/issues/3232","id":188479402,"number":3232,"title":"[distributed]: Data race  when nodes are restarted. ","user":{"login":"hackintoshrao","id":2609511,"avatar_url":"https://avatars.githubusercontent.com/u/2609511?v=3","gravatar_id":"","url":"https://api.github.com/users/hackintoshrao","html_url":"https://github.com/hackintoshrao","followers_url":"https://api.github.com/users/hackintoshrao/followers","following_url":"https://api.github.com/users/hackintoshrao/following{/other_user}","gists_url":"https://api.github.com/users/hackintoshrao/gists{/gist_id}","starred_url":"https://api.github.com/users/hackintoshrao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hackintoshrao/subscriptions","organizations_url":"https://api.github.com/users/hackintoshrao/orgs","repos_url":"https://api.github.com/users/hackintoshrao/repos","events_url":"https://api.github.com/users/hackintoshrao/events{/privacy}","received_events_url":"https://api.github.com/users/hackintoshrao/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-11-10T11:04:13Z","updated_at":"2016-11-10T11:04:13Z","closed_at":null,"body":"<!--- Provide a general summary of the issue in the Title above -->\r\n-  server didn't respond for while but CPU usage was very high (around 700%) when nodes are stopped and started randomly.\r\n- Later when it started most of the nodes reported data race. (binary was built with -race flag)\r\n## Expected Behavior\r\n<!--- If you're describing a bug, tell us what should happen -->\r\n<!--- If you're suggesting a change/improvement, tell us how it should work -->\r\n- Data race shouldnt happen.\r\n## Current Behavior\r\n<!--- If describing a bug, tell us what happens instead of the expected behavior -->\r\n<!--- If suggesting a change/improvement, explain the difference from current behavior -->\r\nhere is the data race log.\r\n\r\n```\r\nInitializing data volume.\r\n[01/08] http://minio1:9199/tmp/rao1 - 218 GiB online\r\n[02/08] http://minio2:9199/tmp/rao1 - 218 GiB online\r\n[03/08] http://minio3:9199/tmp/rao1 - 0 B offline\r\n[04/08] http://minio4:9199/tmp/rao1 - 218 GiB online\r\n[05/08] http://minio5:9199/tmp/rao1 - 218 GiB online\r\n[06/08] http://minio6:9199/tmp/rao1 - 218 GiB online\r\n[07/08] http://minio7:9199/tmp/rao1 - 218 GiB online\r\n[08/08] http://minio8:9199/tmp/rao1 - 218 GiB online\r\n==================\r\nWARNING: DATA RACE\r\nRead at 0x00c420430f50 by goroutine 293:\r\n  github.com/minio/minio/cmd.(*AuthRPCClient).Login()\r\n      /root/work/src/github.com/minio/minio/cmd/auth-rpc-client.go:128 +0x5f\r\n  github.com/minio/minio/cmd.(*AuthRPCClient).Call()\r\n      /root/work/src/github.com/minio/minio/cmd/auth-rpc-client.go:161 +0x45\r\n  github.com/minio/minio/vendor/github.com/minio/dsync.lock.func1()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:172 +0x484\r\n\r\nPrevious write at 0x00c420430f50 by goroutine 61:\r\n  github.com/minio/minio/cmd.(*AuthRPCClient).Login()\r\n      /root/work/src/github.com/minio/minio/cmd/auth-rpc-client.go:149 +0x5b6\r\n  github.com/minio/minio/cmd.(*AuthRPCClient).Call()\r\n      /root/work/src/github.com/minio/minio/cmd/auth-rpc-client.go:161 +0x45\r\n  github.com/minio/minio/vendor/github.com/minio/dsync.lock.func1()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:172 +0x484\r\n\r\nGoroutine 293 (running) created at:\r\ngithub.com/minio/minio/vendor/github.com/minio/dsync.lock()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:191 +0x137\r\n  github.com/minio/minio/vendor/github.com/minio/dsync.(*DRWMutex).lockBlocking()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:120 +0x1ab\r\n  github.com/minio/minio/vendor/github.com/minio/dsync.(*DRWMutex).RLock()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:104 +0x3d\r\n  github.com/minio/minio/cmd.(*nsLockMap).lock()\r\n      /root/work/src/github.com/minio/minio/cmd/namespace-lock.go:140 +0x255\r\n  github.com/minio/minio/cmd.(*lockInstance).RLock()\r\n      /root/work/src/github.com/minio/minio/cmd/namespace-lock.go:290 +0xed\r\n  github.com/minio/minio/cmd.xlObjects.GetBucketInfo()\r\n      /root/work/src/github.com/minio/minio/cmd/xl-v1-bucket.go:177 +0x123\r\n  github.com/minio/minio/cmd.(*xlObjects).GetBucketInfo()\r\n      <autogenerated>:840 +0xfe\r\n  github.com/minio/minio/cmd.isBucketExist()\r\n      /root/work/src/github.com/minio/minio/cmd/object-common.go:282 +0x9e\r\n  github.com/minio/minio/cmd.readBucketPolicyJSON()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:146 +0x7a\r\n  github.com/minio/minio/cmd.readBucketPolicy()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:178 +0x63\r\n  github.com/minio/minio/cmd.loadAllBucketPolicies()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:87 +0x290\r\n  github.com/minio/minio/cmd.initBucketPolicies()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:118 +0x52\r\n  github.com/minio/minio/cmd.newObjectLayer()\r\n      /root/work/src/github.com/minio/minio/cmd/routers.go:69 +0x230\r\n  github.com/minio/minio/cmd.serverMain()\r\n      /root/work/src/github.com/minio/minio/cmd/server-main.go:492 +0xdf6\r\n  github.com/minio/minio/vendor/github.com/minio/cli.Command.Run()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/cli/command.go:117 +0x10d7\r\n  github.com/minio/minio/vendor/github.com/minio/cli.(*App).Run()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/cli/app.go:186 +0x34c\r\n  github.com/minio/minio/vendor/github.com/minio/cli.(*App).RunAndExitOnError()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/cli/app.go:197 +0x70\r\n  github.com/minio/minio/cmd.Main()\r\n      /root/work/src/github.com/minio/minio/cmd/main.go:226 +0x95\r\n  main.main()\r\n      /root/work/src/github.com/minio/minio/main.go:22 +0x2f\r\n\r\nGoroutine 61 (finished) created at:\r\n  github.com/minio/minio/vendor/github.com/minio/dsync.lock()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:191 +0x137\r\n  github.com/minio/minio/vendor/github.com/minio/dsync.(*DRWMutex).lockBlocking()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:120 +0x1ab\r\n  github.com/minio/minio/vendor/github.com/minio/dsync.(*DRWMutex).RLock()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:104 +0x3d\r\n  github.com/minio/minio/cmd.(*nsLockMap).lock()\r\n  /root/work/src/github.com/minio/minio/cmd/namespace-lock.go:140 +0x255\r\n  github.com/minio/minio/cmd.(*lockInstance).RLock()\r\n      /root/work/src/github.com/minio/minio/cmd/namespace-lock.go:290 +0xed\r\n  github.com/minio/minio/cmd.xlObjects.GetBucketInfo()\r\n      /root/work/src/github.com/minio/minio/cmd/xl-v1-bucket.go:177 +0x123\r\n  github.com/minio/minio/cmd.(*xlObjects).GetBucketInfo()\r\n      <autogenerated>:840 +0xfe\r\n  github.com/minio/minio/cmd.isBucketExist()\r\n      /root/work/src/github.com/minio/minio/cmd/object-common.go:282 +0x9e\r\n  github.com/minio/minio/cmd.readBucketPolicyJSON()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:146 +0x7a\r\n  github.com/minio/minio/cmd.readBucketPolicy()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:178 +0x63\r\n  github.com/minio/minio/cmd.loadAllBucketPolicies()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:87 +0x290\r\n  github.com/minio/minio/cmd.initBucketPolicies()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:118 +0x52\r\n  github.com/minio/minio/cmd.newObjectLayer()\r\n      /root/work/src/github.com/minio/minio/cmd/routers.go:69 +0x230\r\n  github.com/minio/minio/cmd.serverMain()\r\n      /root/work/src/github.com/minio/minio/cmd/server-main.go:492 +0xdf6\r\n  github.com/minio/minio/vendor/github.com/minio/cli.Command.Run()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/cli/command.go:117 +0x10d7\r\n  github.com/minio/minio/vendor/github.com/minio/cli.(*App).Run()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/cli/app.go:186 +0x34c\r\n  github.com/minio/minio/vendor/github.com/minio/cli.(*App).RunAndExitOnError()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/cli/app.go:197 +0x70\r\n  github.com/minio/minio/cmd.Main()\r\n      /root/work/src/github.com/minio/minio/cmd/main.go:226 +0x95\r\n  main.main()\r\n      /root/work/src/github.com/minio/minio/main.go:22 +0x2f\r\n==================\r\n==================\r\nWARNING: DATA RACE\r\nRead at 0x00c420430f58 by goroutine 293:\r\n  github.com/minio/minio/cmd.(*AuthRPCClient).Call()\r\n      /root/work/src/github.com/minio/minio/cmd/auth-rpc-client.go:163 +0x9f\r\n  github.com/minio/minio/vendor/github.com/minio/dsync.lock.func1()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:172 +0x484\r\n\r\nPrevious write at 0x00c420430f58 by goroutine 61:\r\n  github.com/minio/minio/cmd.(*AuthRPCClient).Login()\r\n      /root/work/src/github.com/minio/minio/cmd/auth-rpc-client.go:147 +0x511\r\n  github.com/minio/minio/cmd.(*AuthRPCClient).Call()\r\n      /root/work/src/github.com/minio/minio/cmd/auth-rpc-client.go:161 +0x45\r\n  github.com/minio/minio/vendor/github.com/minio/dsync.lock.func1()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:172 +0x484\r\n\r\nGoroutine 293 (running) created at:\r\n  github.com/minio/minio/vendor/github.com/minio/dsync.lock()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:191 +0x137\r\n github.com/minio/minio/vendor/github.com/minio/dsync.(*DRWMutex).lockBlocking()                                                                                           [25/1990]\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:120 +0x1ab\r\n  github.com/minio/minio/vendor/github.com/minio/dsync.(*DRWMutex).RLock()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:104 +0x3d\r\n  github.com/minio/minio/cmd.(*nsLockMap).lock()\r\n      /root/work/src/github.com/minio/minio/cmd/namespace-lock.go:140 +0x255\r\n  github.com/minio/minio/cmd.(*lockInstance).RLock()\r\n      /root/work/src/github.com/minio/minio/cmd/namespace-lock.go:290 +0xed\r\n  github.com/minio/minio/cmd.xlObjects.GetBucketInfo()\r\n      /root/work/src/github.com/minio/minio/cmd/xl-v1-bucket.go:177 +0x123\r\n  github.com/minio/minio/cmd.(*xlObjects).GetBucketInfo()\r\n      <autogenerated>:840 +0xfe\r\n  github.com/minio/minio/cmd.isBucketExist()\r\n      /root/work/src/github.com/minio/minio/cmd/object-common.go:282 +0x9e\r\n  github.com/minio/minio/cmd.readBucketPolicyJSON()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:146 +0x7a\r\n  github.com/minio/minio/cmd.readBucketPolicy()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:178 +0x63\r\n  github.com/minio/minio/cmd.loadAllBucketPolicies()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:87 +0x290\r\n  github.com/minio/minio/cmd.initBucketPolicies()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:118 +0x52\r\n  github.com/minio/minio/cmd.newObjectLayer()\r\n      /root/work/src/github.com/minio/minio/cmd/routers.go:69 +0x230\r\n  github.com/minio/minio/cmd.serverMain()\r\n      /root/work/src/github.com/minio/minio/cmd/server-main.go:492 +0xdf6\r\n  github.com/minio/minio/vendor/github.com/minio/cli.Command.Run()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/cli/command.go:117 +0x10d7\r\n  github.com/minio/minio/vendor/github.com/minio/cli.(*App).Run()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/cli/app.go:186 +0x34c\r\n  github.com/minio/minio/vendor/github.com/minio/cli.(*App).RunAndExitOnError()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/cli/app.go:197 +0x70\r\n  github.com/minio/minio/cmd.Main()\r\n      /root/work/src/github.com/minio/minio/cmd/main.go:226 +0x95\r\n  main.main()\r\n      /root/work/src/github.com/minio/minio/main.go:22 +0x2f\r\n\r\nGoroutine 61 (finished) created at:\r\n  github.com/minio/minio/vendor/github.com/minio/dsync.lock()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:191 +0x137\r\n  github.com/minio/minio/vendor/github.com/minio/dsync.(*DRWMutex).lockBlocking()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:120 +0x1ab\r\n  github.com/minio/minio/vendor/github.com/minio/dsync.(*DRWMutex).RLock()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:104 +0x3d\r\n  github.com/minio/minio/cmd.(*nsLockMap).lock()\r\n      /root/work/src/github.com/minio/minio/cmd/namespace-lock.go:140 +0x255\r\n  github.com/minio/minio/cmd.(*lockInstance).RLock()\r\n      /root/work/src/github.com/minio/minio/cmd/namespace-lock.go:290 +0xed\r\n  github.com/minio/minio/cmd.xlObjects.GetBucketInfo()\r\n      /root/work/src/github.com/minio/minio/cmd/xl-v1-bucket.go:177 +0x123\r\n  github.com/minio/minio/cmd.(*xlObjects).GetBucketInfo()\r\n      <autogenerated>:840 +0xfe\r\n  github.com/minio/minio/cmd.isBucketExist()\r\n      /root/work/src/github.com/minio/minio/cmd/object-common.go:282 +0x9e\r\n  github.com/minio/minio/cmd.readBucketPolicyJSON()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:146 +0x7a\r\n  github.com/minio/minio/cmd.readBucketPolicy()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:178 +0x63\r\n  github.com/minio/minio/cmd.loadAllBucketPolicies()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:87 +0x290\r\n  github.com/minio/minio/cmd.initBucketPolicies()\r\n      /root/work/src/github.com/minio/minio/cmd/bucket-policy.go:118 +0x52\r\n  github.com/minio/minio/cmd.newObjectLayer()\r\n      /root/work/src/github.com/minio/minio/cmd/routers.go:69 +0x230\r\n  github.com/minio/minio/cmd.serverMain()\r\n      /root/work/src/github.com/minio/minio/cmd/server-main.go:492 +0xdf6\r\n  github.com/minio/minio/vendor/github.com/minio/cli.Command.Run()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/cli/command.go:117 +0x10d7\r\n  github.com/minio/minio/vendor/github.com/minio/cli.(*App).Run()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/cli/app.go:186 +0x34c\r\n  github.com/minio/minio/vendor/github.com/minio/cli.(*App).RunAndExitOnError()\r\n      /root/work/src/github.com/minio/minio/vendor/github.com/minio/cli/app.go:197 +0x70\r\n  github.com/minio/minio/cmd.Main()\r\n      /root/work/src/github.com/minio/minio/cmd/main.go:226 +0x95\r\n  main.main()\r\n      /root/work/src/github.com/minio/minio/main.go:22 +0x2f\r\n==================\r\n```\r\n\r\n## Possible Solution\r\n<!--- Not obligatory, but suggest a fix/reason for the bug, -->\r\n<!--- or ideas how to implement the addition or change -->\r\nNot sure.\r\n## Steps to Reproduce (for bugs)\r\n<!--- Provide a link to a live example, or an unambiguous set of steps to -->\r\n<!--- reproduce this bug. Include code to reproduce, if relevant -->\r\n1. Build binary with `-race`. \r\n2. Start the server on 8 nodes. Randomly kill them and start again\r\n## Context\r\n<!--- How has this issue affected you? What are you trying to accomplish? -->\r\n<!--- Providing context helps us come up with a solution that is most useful in the real world -->\r\nTesting with nodes being restarted.\r\n## Your Environment\r\n<!--- Include as many relevant details about the environment you experienced the bug in -->\r\n* Version used: Latest master 2f373684f5ce9310fd78677b2d76d1d5ac9cbce8\r\n\r\n"}}
{"actor":{"display_login":"hackintoshrao"}}