{"action":"created","issue":{"url":"https://api.github.com/repos/EventStore/EventStore/issues/940","repository_url":"https://api.github.com/repos/EventStore/EventStore","labels_url":"https://api.github.com/repos/EventStore/EventStore/issues/940/labels{/name}","comments_url":"https://api.github.com/repos/EventStore/EventStore/issues/940/comments","events_url":"https://api.github.com/repos/EventStore/EventStore/issues/940/events","html_url":"https://github.com/EventStore/EventStore/issues/940","id":159159682,"number":940,"title":"Client recieves 'ReadStreamEventsCompleted' message with empty event list when running es cluster","user":{"login":"msemys","id":7299070,"avatar_url":"https://avatars.githubusercontent.com/u/7299070?v=3","gravatar_id":"","url":"https://api.github.com/users/msemys","html_url":"https://github.com/msemys","followers_url":"https://api.github.com/users/msemys/followers","following_url":"https://api.github.com/users/msemys/following{/other_user}","gists_url":"https://api.github.com/users/msemys/gists{/gist_id}","starred_url":"https://api.github.com/users/msemys/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msemys/subscriptions","organizations_url":"https://api.github.com/users/msemys/orgs","repos_url":"https://api.github.com/users/msemys/repos","events_url":"https://api.github.com/users/msemys/events{/privacy}","received_events_url":"https://api.github.com/users/msemys/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2016-06-08T13:10:23Z","updated_at":"2016-06-10T08:27:04Z","closed_at":"2016-06-09T12:39:59Z","body":"I'm running a test that writes N events (where N is random number greater than 100 and less than 500)  to stream and then reads and asserts them. This write/read action is repeated 10000 times (in for loop).\r\n\r\nWhen running EventStore v3.6.3 cluster, sometimes (with 10000 iterations - always) the 'ReadStreamEventsBackward' operation returns 'ReadStreamEventsCompleted' message with no events. The received message has status, nextEventNumber, lastEventNumber, isEndOfStream flag set, but events field is empty list.\r\n\r\n**machine details**:\r\n- ubuntu 14.04 x64\r\n- ram 8gb\r\n- cpu i7-4770K CPU @ 3.50GHz x 8\r\n- ssd (samsung 840 pro)\r\n\r\n**es cluster nodes started with these commands (same machine)**:\r\n````sh\r\n./run-node.sh --mem-db --log log/node1 --int-ip 127.0.0.1 --ext-ip 127.0.0.1 --int-tcp-port=1111 --ext-tcp-port=1112 --int-http-port=1113 --ext-http-port=1114 --cluster-size=3 --discover-via-dns=false --gossip-seed=127.0.0.1:2113,127.0.0.1:3113\r\n\r\n./run-node.sh --mem-db --log log/node2 --int-ip 127.0.0.1 --ext-ip 127.0.0.1 --int-tcp-port=2111 --ext-tcp-port=2112 --int-http-port=2113 --ext-http-port=2114 --cluster-size=3 --discover-via-dns=false --gossip-seed=127.0.0.1:1113,127.0.0.1:3113\r\n\r\n./run-node.sh --mem-db --log log/node3 --int-ip 127.0.0.1 --ext-ip 127.0.0.1 --int-tcp-port=3111 --ext-tcp-port=3112 --int-http-port=3113 --ext-http-port=3114 --cluster-size=3 --discover-via-dns=false --gossip-seed=127.0.0.1:1113,127.0.0.1:2113\r\n```\r\n\r\n**es logs**:\r\n- [node1.txt](https://github.com/EventStore/EventStore/files/304779/node1.txt)\r\n- [node2.txt](https://github.com/EventStore/EventStore/files/304781/node2.txt)\r\n- [node3.txt](https://github.com/EventStore/EventStore/files/304785/node3.txt)\r\n\r\n\r\n**notes**:\r\n- issue occurs with es cluster only (with single node ok)\r\n- it is reproducible with java8/netty (esjc 1.4.0) and scala/akka (es.jvm 2.1.2) clients\r\n- test passed when I add delay (thread.sleep 50ms) between write and read operations in test\r\n\r\n\r\n**reproduce with esjc 1.4.0 client**\r\n```java\r\n@Test\r\npublic void readsWrittenEvents() {\r\n    final EventStore eventstore = EventStoreBuilder.newBuilder()\r\n        .clusterNodeDiscoveryFromGossipSeeds(asList(\r\n            new InetSocketAddress(\"127.0.0.1\", 1113),\r\n            new InetSocketAddress(\"127.0.0.1\", 2113),\r\n            new InetSocketAddress(\"127.0.0.1\", 3113)))\r\n        .clusterNodeGossipTimeout(Duration.ofSeconds(1))\r\n        .clusterNodeUnlimitedDiscoverAttempts()\r\n        .userCredentials(\"admin\", \"changeit\")\r\n        .operationTimeout(Duration.ofSeconds(5))\r\n        .heartbeatInterval(Duration.ofSeconds(2))\r\n        .heartbeatTimeout(Duration.ofSeconds(10))\r\n        .maxClientReconnections(2)\r\n        .requireMasterEnabled()\r\n        .build();\r\n\r\n    final String stream = \"reads-written-events-\" + UUID.randomUUID().toString();\r\n\r\n    range(0, 10000).forEach(i -> {\r\n        final int count = ThreadLocalRandom.current().nextInt(100, 500);\r\n\r\n        List<EventData> events = range(0, count).mapToObj(n -> EventData.newBuilder().type(\"test\").build()).collect(toList());\r\n\r\n        WriteResult writeResult = eventstore.appendToStream(stream, ExpectedVersion.any(), events).join();\r\n\r\n        StreamEventsSlice readResult = eventstore.readStreamEventsBackward(stream, writeResult.nextExpectedVersion, count, false).join();\r\n\r\n        assertEquals(SliceReadStatus.Success, readResult.status);\r\n        assertEquals(writeResult.nextExpectedVersion, readResult.fromEventNumber);\r\n        assertEquals(count, readResult.events.size());\r\n\r\n        range(0, count).forEach(n ->\r\n            assertEquals(events.get(n).eventId, readResult.events.get(count - n - 1).originalEvent().eventId)\r\n        );\r\n    });\r\n}\r\n```\r\n\r\n**reproduce with es.jvm 2.1.2 client**\r\n```java\r\n@Test\r\npublic void readsWrittenEvents() {\r\n    final EsConnection connection = EsConnectionFactory.create(ActorSystem.create(), new SettingsBuilder()\r\n        .cluster(new ClusterSettingsBuilder()\r\n            .gossipSeeds(asScalaBuffer(asList(\r\n                new InetSocketAddress(\"127.0.0.1\", 1113),\r\n                new InetSocketAddress(\"127.0.0.1\", 2113),\r\n                new InetSocketAddress(\"127.0.0.1\", 3113))))\r\n            .gossipTimeout(1, SECONDS)\r\n            .build())\r\n        .defaultCredentials(\"admin\", \"changeit\")\r\n        .operationTimeout(5, SECONDS)\r\n        .heartbeatInterval(2, SECONDS)\r\n        .heartbeatTimeout(10, SECONDS)\r\n        .maxReconnections(2)\r\n        .requireMaster(true)\r\n        .build());\r\n\r\n    final String stream = \"reads-written-events-\" + UUID.randomUUID().toString();\r\n\r\n    range(0, 10000).forEach(i -> {\r\n        final int count = ThreadLocalRandom.current().nextInt(100, 500);\r\n\r\n        List<EventData> events = range(0, count).mapToObj(n -> new EventDataBuilder(\"test\").eventId(UUID.randomUUID()).build()).collect(toList());\r\n\r\n        WriteResult writeResult = await(connection.writeEvents(stream, ExpectedVersion.Any$.MODULE$, events, null));\r\n\r\n        ReadStreamEventsCompleted readResult = await(connection.readStreamEventsBackward(stream, new EventNumber.Exact(writeResult.nextExpectedVersion().value()), count, false, null));\r\n\r\n        assertEquals(count, readResult.events().size());\r\n\r\n        range(0, count).forEach(n ->\r\n            assertEquals(events.get(n).eventId(), readResult.eventsJava().get(count - n - 1).data().eventId())\r\n        );\r\n    });\r\n}\r\n\r\nprivate static <T> T await(Future<T> future) {\r\n    try {\r\n        return Await.result(future, Duration.apply(15, SECONDS));\r\n    } catch (Exception e) {\r\n        throw new RuntimeException(e);\r\n    }\r\n}\r\n```\r\n\r\ntests fails when asserting read events count:\r\n\r\n```\r\n>>> assertEquals(count, readResult.events.size());\r\n\r\njava.lang.AssertionError: \r\nExpected :177\r\nActual   :0\r\n```"},"comment":{"url":"https://api.github.com/repos/EventStore/EventStore/issues/comments/225122502","html_url":"https://github.com/EventStore/EventStore/issues/940#issuecomment-225122502","issue_url":"https://api.github.com/repos/EventStore/EventStore/issues/940","id":225122502,"user":{"login":"msemys","id":7299070,"avatar_url":"https://avatars.githubusercontent.com/u/7299070?v=3","gravatar_id":"","url":"https://api.github.com/users/msemys","html_url":"https://github.com/msemys","followers_url":"https://api.github.com/users/msemys/followers","following_url":"https://api.github.com/users/msemys/following{/other_user}","gists_url":"https://api.github.com/users/msemys/gists{/gist_id}","starred_url":"https://api.github.com/users/msemys/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msemys/subscriptions","organizations_url":"https://api.github.com/users/msemys/orgs","repos_url":"https://api.github.com/users/msemys/repos","events_url":"https://api.github.com/users/msemys/events{/privacy}","received_events_url":"https://api.github.com/users/msemys/received_events","type":"User","site_admin":false},"created_at":"2016-06-10T08:27:04Z","updated_at":"2016-06-10T08:27:04Z","body":"I've updated test that uses expected version for write operation (require master is also set).\r\n\r\n```java\r\n@Test\r\npublic void readsWrittenEvents() {\r\n    final EventStore eventstore = EventStoreBuilder.newBuilder()\r\n        .clusterNodeDiscoveryFromGossipSeeds(asList(\r\n            new InetSocketAddress(\"127.0.0.1\", 1113),\r\n            new InetSocketAddress(\"127.0.0.1\", 2113),\r\n            new InetSocketAddress(\"127.0.0.1\", 3113)))\r\n        .clusterNodeGossipTimeout(Duration.ofSeconds(1))\r\n        .clusterNodeUnlimitedDiscoverAttempts()\r\n        .userCredentials(\"admin\", \"changeit\")\r\n        .operationTimeout(Duration.ofSeconds(5))\r\n        .heartbeatInterval(Duration.ofSeconds(2))\r\n        .heartbeatTimeout(Duration.ofSeconds(10))\r\n        .maxClientReconnections(2)\r\n        .requireMasterEnabled()\r\n        .build();\r\n\r\n    final String stream = \"reads-written-events-\" + UUID.randomUUID().toString();\r\n\r\n    ExpectedVersion expectedVersion = ExpectedVersion.noStream();\r\n\r\n    for (int i = 0; i < 10000; i++) {\r\n        final int count = ThreadLocalRandom.current().nextInt(100, 500);\r\n\r\n        List<EventData> events = range(0, count).mapToObj(n -> EventData.newBuilder().type(\"test\").build()).collect(toList());\r\n\r\n        WriteResult writeResult = eventstore.appendToStream(stream, expectedVersion, events).join();\r\n\r\n        StreamEventsSlice readResult = eventstore.readStreamEventsForward(stream, writeResult.nextExpectedVersion - count + 1, count, false).join();\r\n\r\n        assertEquals(SliceReadStatus.Success, readResult.status);\r\n        assertEquals(count, readResult.events.size());\r\n\r\n        range(0, count).forEach(n ->\r\n            assertEquals(events.get(n).eventId, readResult.events.get(n).originalEvent().eventId)\r\n        );\r\n\r\n        expectedVersion = ExpectedVersion.of(writeResult.nextExpectedVersion);\r\n    }\r\n}\r\n```\r\n\r\nwrites succeeds (no WrongExpectedVersion in WriteEventsCompleted message), but test sometimes fail when asserting read event count (the same as it was when using ```ExpectedVersion.any()```)\r\n\r\nI think that this should be reproducible with .net client as well."}}
{"actor":{"display_login":"msemys"}}