{"action":"created","issue":{"url":"https://api.github.com/repos/nats-io/nats-streaming-server/issues/474","repository_url":"https://api.github.com/repos/nats-io/nats-streaming-server","labels_url":"https://api.github.com/repos/nats-io/nats-streaming-server/issues/474/labels{/name}","comments_url":"https://api.github.com/repos/nats-io/nats-streaming-server/issues/474/comments","events_url":"https://api.github.com/repos/nats-io/nats-streaming-server/issues/474/events","html_url":"https://github.com/nats-io/nats-streaming-server/issues/474","id":297252946,"number":474,"title":"Segmentation violation in cluster mode and latest docker container","user":{"login":"sega-yarkin","id":1433049,"avatar_url":"https://avatars0.githubusercontent.com/u/1433049?v=4","gravatar_id":"","url":"https://api.github.com/users/sega-yarkin","html_url":"https://github.com/sega-yarkin","followers_url":"https://api.github.com/users/sega-yarkin/followers","following_url":"https://api.github.com/users/sega-yarkin/following{/other_user}","gists_url":"https://api.github.com/users/sega-yarkin/gists{/gist_id}","starred_url":"https://api.github.com/users/sega-yarkin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sega-yarkin/subscriptions","organizations_url":"https://api.github.com/users/sega-yarkin/orgs","repos_url":"https://api.github.com/users/sega-yarkin/repos","events_url":"https://api.github.com/users/sega-yarkin/events{/privacy}","received_events_url":"https://api.github.com/users/sega-yarkin/received_events","type":"User","site_admin":false},"labels":[{"id":347049928,"url":"https://api.github.com/repos/nats-io/nats-streaming-server/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"closed","locked":false,"assignee":{"login":"kozlovic","id":13736691,"avatar_url":"https://avatars3.githubusercontent.com/u/13736691?v=4","gravatar_id":"","url":"https://api.github.com/users/kozlovic","html_url":"https://github.com/kozlovic","followers_url":"https://api.github.com/users/kozlovic/followers","following_url":"https://api.github.com/users/kozlovic/following{/other_user}","gists_url":"https://api.github.com/users/kozlovic/gists{/gist_id}","starred_url":"https://api.github.com/users/kozlovic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kozlovic/subscriptions","organizations_url":"https://api.github.com/users/kozlovic/orgs","repos_url":"https://api.github.com/users/kozlovic/repos","events_url":"https://api.github.com/users/kozlovic/events{/privacy}","received_events_url":"https://api.github.com/users/kozlovic/received_events","type":"User","site_admin":false},"assignees":[{"login":"kozlovic","id":13736691,"avatar_url":"https://avatars3.githubusercontent.com/u/13736691?v=4","gravatar_id":"","url":"https://api.github.com/users/kozlovic","html_url":"https://github.com/kozlovic","followers_url":"https://api.github.com/users/kozlovic/followers","following_url":"https://api.github.com/users/kozlovic/following{/other_user}","gists_url":"https://api.github.com/users/kozlovic/gists{/gist_id}","starred_url":"https://api.github.com/users/kozlovic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kozlovic/subscriptions","organizations_url":"https://api.github.com/users/kozlovic/orgs","repos_url":"https://api.github.com/users/kozlovic/repos","events_url":"https://api.github.com/users/kozlovic/events{/privacy}","received_events_url":"https://api.github.com/users/kozlovic/received_events","type":"User","site_admin":false}],"milestone":null,"comments":20,"created_at":"2018-02-14T21:33:40Z","updated_at":"2018-03-06T22:21:31Z","closed_at":"2018-02-20T20:03:57Z","author_association":"NONE","body":"Hi guys!\r\nI've just tested latest version of NATS Streaming in cluster mode with 3 nodes and got `panic: runtime error: invalid memory address or nil pointer dereference`.\r\n\r\nMy steps:\r\n1) Create 3 data directories (`srv1`, `srv2`, `srv3`)\r\n2) Put simple `nats.conf` in each data directory (ports end with 1, 2 and 3 respectively):\r\n```\r\nport: 4231\r\nhttp_port: 8231\r\nstore: \"file\"\r\ndir: \"/data\"\r\ncluster {\r\n  host: '10.10.0.100'\r\n  port: 5231\r\n  bootstrap: true\r\n  raft_logging: true\r\n  routes = [\r\n    nats-route://10.10.0.100:5232\r\n    nats-route://10.10.0.100:5233\r\n  ]\r\n}\r\n```\r\n3) Start 3 docker containers:\r\n`docker container run -it --name nats1 -v ${PWD}/srv1:/data --network=host nats-streaming -c /data/nats.conf`\r\n4) Do some simple tests to be sure everything is working.\r\n5) Start 3 consumers (just commented out printing every messages):\r\n```\r\nstan-sub -s nats://10.10.0.100:4231 -id 1 --qgroup qgtest -all test\r\nstan-sub -s nats://10.10.0.100:4232 -id 2 --qgroup qgtest -all test\r\nstan-sub -s nats://10.10.0.100:4233 -id 3 --qgroup qgtest -all test\r\n```\r\n6) Start data producer:\r\n```\r\nstan-bench -s nats://10.10.0.100:4231 -np 1 -ns 0 -n 10000000 -ms 1000 -a test\r\n```\r\n\r\nAfter several minutes second and third containers started printing errors:\r\n```\r\n...\r\n[1] 2018/02/14 20:32:31.100452 [ERR] STREAM: [Client:benchmark-pub-1] Error replicating message for subject \"_STAN.pub.test-cluster.test\": timed out enqueuing operation\r\n[1] 2018/02/14 20:32:31.100464 [ERR] STREAM: [Client:benchmark-pub-97] Error replicating message for subject \"_STAN.pub.test-cluster.test\": timed out enqueuing operation\r\n[1] 2018/02/14 20:32:31.100475 [ERR] STREAM: [Client:benchmark-pub-94] Error replicating message for subject \"_STAN.pub.test-cluster.test\": timed out enqueuing operation\r\n[1] 2018/02/14 20:32:31.100485 [ERR] STREAM: [Client:benchmark-pub-97] Error replicating message for subject \"_STAN.pub.test-cluster.test\": timed out enqueuing operation\r\n...\r\n```\r\n\r\nand then:\r\n```\r\n[1] 2018/02/14 20:32:27.598057 [ERR] STREAM: [Client:benchmark-pub-85] Error replicating message for subject \"_STAN.pub.test-cluster.test\": timed out enqueuing operation\r\npanic: runtime error: invalid memory address or nil pointer dereference\r\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x87e359]\r\n\r\ngoroutine 70 [running]:\r\ngithub.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).removeFirstMsg(0xc42009a340, 0x0, 0xc42ec9af00)\r\n\t/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2647 +0x69\r\ngithub.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).enforceLimits(0xc42009a340, 0x1e0001, 0xc4531b12c0, 0xc447f7b801)\r\n\t/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2589 +0xa2\r\ngithub.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).Store(0xc42009a340, 0xc4531b12c0, 0x0, 0x0, 0x0)\r\n\t/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2482 +0xb99\r\ngithub.com/nats-io/nats-streaming-server/server.(*StanServer).Apply(0xc4201ea000, 0xc450f849a0, 0xdfe320, 0x3)\r\n\t/go/src/github.com/nats-io/nats-streaming-server/server/server.go:4498 +0x176\r\ngithub.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc44b13b1e0)\r\n\t/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/fsm.go:57 +0x17b\r\ngithub.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).runFSM(0xc4200e1340)\r\n\t/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/fsm.go:120 +0x31e\r\ngithub.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).(github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.runFSM)-fm()\r\n\t/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/api.go:506 +0x2a\r\ngithub.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc4200e1340, 0xc420180500)\r\n\t/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/state.go:146 +0x53\r\ncreated by github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*raftState).goFunc\r\n\t/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/state.go:144 +0x66\r\n```\r\n\r\nAfter restarting failed containers:\r\n```\r\n$ docker container start -ai nats2\r\n[1] 2018/02/14 20:34:30.879042 [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.8.0-beta\r\n[1] 2018/02/14 20:34:30.879082 [INF] STREAM: ServerID: PIWf5lutLWcVp9fYVr2WFY\r\n[1] 2018/02/14 20:34:30.879087 [INF] STREAM: Go version: go1.9.3\r\n[1] 2018/02/14 20:34:30.879323 [INF] Starting nats-server version 1.0.4\r\n[1] 2018/02/14 20:34:30.879554 [INF] Starting http monitor on 0.0.0.0:8232\r\n[1] 2018/02/14 20:34:30.879671 [INF] Listening for client connections on 0.0.0.0:4232\r\n[1] 2018/02/14 20:34:30.879685 [INF] Server is ready\r\n[1] 2018/02/14 20:34:30.880552 [INF] Listening for route connections on 10.10.0.100:5232\r\n[1] 2018/02/14 20:34:30.906597 [INF] STREAM: Recovering the state...\r\n[1] 2018/02/14 20:34:31.373113 [INF] STREAM: Recovered 1 channel(s)\r\n[1] 2018/02/14 20:34:33.298825 [INF] STREAM: raft: Initial configuration (index=1): [{Suffrage:Voter ID:ib2LAFTTccvyusMmcYgM9D Address:test-cluster.ib2LAFTTccvyusMmcYgM9D.stan}]\r\n[1] 2018/02/14 20:34:33.298939 [INF] STREAM: raft: Node at test-cluster.ib2LAFTTccvyusMmcYgM9D.stan [Follower] entering Follower state (Leader: \"\")\r\n[1] 2018/02/14 20:34:33.300337 [INF] STREAM: Message store is RAFT_FILE\r\n[1] 2018/02/14 20:34:33.300415 [INF] STREAM: ---------- Store Limits ----------\r\n[1] 2018/02/14 20:34:33.300425 [INF] STREAM: Channels:                  100 *\r\n[1] 2018/02/14 20:34:33.300431 [INF] STREAM: --------- Channels Limits --------\r\n[1] 2018/02/14 20:34:33.300436 [INF] STREAM:   Subscriptions:          1000 *\r\n[1] 2018/02/14 20:34:33.300443 [INF] STREAM:   Messages     :       1000000 *\r\n[1] 2018/02/14 20:34:33.300449 [INF] STREAM:   Bytes        :     976.56 MB *\r\n[1] 2018/02/14 20:34:33.300456 [INF] STREAM:   Age          :     unlimited *\r\n[1] 2018/02/14 20:34:33.300464 [INF] STREAM: ----------------------------------\r\n[1] 2018/02/14 20:34:35.169644 [INF] STREAM: raft: Heartbeat timeout from \"\" reached, starting election\r\n[1] 2018/02/14 20:34:35.169694 [INF] STREAM: raft: Node at test-cluster.ib2LAFTTccvyusMmcYgM9D.stan [Candidate] entering Candidate state in term 3\r\n[1] 2018/02/14 20:34:35.169961 [INF] STREAM: raft: Election won. Tally: 1\r\n[1] 2018/02/14 20:34:35.169977 [INF] STREAM: raft: Node at test-cluster.ib2LAFTTccvyusMmcYgM9D.stan [Leader] entering Leader state\r\n[1] 2018/02/14 20:34:35.170079 [INF] STREAM: server became leader, performing leader promotion actions\r\n[1] 2018/02/14 20:34:38.276052 [INF] STREAM: WARNING: Reached limits for store \"test\" (msgs=963311/1000000 bytes=976.56 MB/976.56 MB), dropping old messages to make room for new ones\r\npanic: runtime error: invalid memory address or nil pointer dereference\r\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x87e359]\r\n\r\ngoroutine 92 [running]:\r\ngithub.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).removeFirstMsg(0xc42023a000, 0x0, 0xc4203e1600)\r\n\t/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2647 +0x69\r\ngithub.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).enforceLimits(0xc42023a000, 0x1e0001, 0xc42172aa50, 0xc422016a01)\r\n\t/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2589 +0xa2\r\ngithub.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).Store(0xc42023a000, 0xc42172aa50, 0x0, 0x0, 0x0)\r\n\t/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2482 +0xb99\r\ngithub.com/nats-io/nats-streaming-server/server.(*StanServer).Apply(0xc4201ee000, 0xc4203354a0, 0xdfe320, 0xbe99453790618e89)\r\n\t/go/src/github.com/nats-io/nats-streaming-server/server/server.go:4498 +0x176\r\ngithub.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc4202ec2d0)\r\n\t/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/fsm.go:57 +0x17b\r\ngithub.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).runFSM(0xc4200d0000)\r\n\t/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/fsm.go:120 +0x31e\r\ngithub.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).(github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.runFSM)-fm()\r\n\t/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/api.go:506 +0x2a\r\ngithub.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc4200d0000, 0xc4202ec150)\r\n\t/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/state.go:146 +0x53\r\ncreated by github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*raftState).goFunc\r\n\t/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/state.go:144 +0x66\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/nats-io/nats-streaming-server/issues/comments/370950169","html_url":"https://github.com/nats-io/nats-streaming-server/issues/474#issuecomment-370950169","issue_url":"https://api.github.com/repos/nats-io/nats-streaming-server/issues/474","id":370950169,"user":{"login":"sega-yarkin","id":1433049,"avatar_url":"https://avatars0.githubusercontent.com/u/1433049?v=4","gravatar_id":"","url":"https://api.github.com/users/sega-yarkin","html_url":"https://github.com/sega-yarkin","followers_url":"https://api.github.com/users/sega-yarkin/followers","following_url":"https://api.github.com/users/sega-yarkin/following{/other_user}","gists_url":"https://api.github.com/users/sega-yarkin/gists{/gist_id}","starred_url":"https://api.github.com/users/sega-yarkin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sega-yarkin/subscriptions","organizations_url":"https://api.github.com/users/sega-yarkin/orgs","repos_url":"https://api.github.com/users/sega-yarkin/repos","events_url":"https://api.github.com/users/sega-yarkin/events{/privacy}","received_events_url":"https://api.github.com/users/sega-yarkin/received_events","type":"User","site_admin":false},"created_at":"2018-03-06T22:21:31Z","updated_at":"2018-03-06T22:21:31Z","author_association":"NONE","body":"@kozlovic Yes, I ran test during last weekends with raft logs path and I wasn't able to reproduce it, I don't know how it was possible before (it definitely was the latest commit). I will try to run tests without log path and message here if I be able to reproduce the panic. Thank you!"}}
{"actor":{"display_login":"sega-yarkin"}}