{"action":"created","issue":{"url":"https://api.github.com/repos/neo4j/neo4j/issues/6087","labels_url":"https://api.github.com/repos/neo4j/neo4j/issues/6087/labels{/name}","comments_url":"https://api.github.com/repos/neo4j/neo4j/issues/6087/comments","events_url":"https://api.github.com/repos/neo4j/neo4j/issues/6087/events","html_url":"https://github.com/neo4j/neo4j/issues/6087","id":121883282,"number":6087,"title":"Can't share index-backed ResourceIterable between transactions","user":{"login":"bachmanm","id":629665,"avatar_url":"https://avatars.githubusercontent.com/u/629665?v=3","gravatar_id":"","url":"https://api.github.com/users/bachmanm","html_url":"https://github.com/bachmanm","followers_url":"https://api.github.com/users/bachmanm/followers","following_url":"https://api.github.com/users/bachmanm/following{/other_user}","gists_url":"https://api.github.com/users/bachmanm/gists{/gist_id}","starred_url":"https://api.github.com/users/bachmanm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bachmanm/subscriptions","organizations_url":"https://api.github.com/users/bachmanm/orgs","repos_url":"https://api.github.com/users/bachmanm/repos","events_url":"https://api.github.com/users/bachmanm/events{/privacy}","received_events_url":"https://api.github.com/users/bachmanm/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/neo4j/neo4j/labels/2.3","name":"2.3","color":"5319e7"},{"url":"https://api.github.com/repos/neo4j/neo4j/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/neo4j/neo4j/labels/kernel","name":"kernel","color":"0052cc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-12-12T21:58:32Z","updated_at":"2016-01-19T09:15:37Z","closed_at":null,"body":"Let's say we're trying to add `SecondLabel` to every node with `FirstLabel`. What's the right way of doing this on a very large database? What we have been doing is something along the lines in the test below. However, any number above 131,000 nodes with `FirstLabel` fails the test with \r\n\r\n```\r\norg.apache.lucene.store.AlreadyClosedException: this IndexReader is closed\r\n\tat org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:245)\r\n\tat org.apache.lucene.index.IndexReader.getSequentialSubReaders(IndexReader.java:1602)\r\n\tat org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:78)\r\n\tat org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:87)\r\n\tat org.apache.lucene.util.ReaderUtil$Gather.run(ReaderUtil.java:70)\r\n\tat org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:53)\r\n\tat org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:199)\r\n\tat org.apache.lucene.search.Searcher.createNormalizedWeight(Searcher.java:168)\r\n\tat org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:664)\r\n\tat org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:319)\r\n\tat org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:305)\r\n\tat org.neo4j.kernel.api.impl.index.PageOfRangesIterator.fetchNextOrNull(PageOfRangesIterator.java:67)\r\n\tat org.neo4j.kernel.api.impl.index.PageOfRangesIterator.fetchNextOrNull(PageOfRangesIterator.java:35)\r\n\tat org.neo4j.helpers.collection.PrefetchingIterator.peek(PrefetchingIterator.java:60)\r\n\tat org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:46)\r\n\tat org.neo4j.collection.primitive.PrimitiveLongCollections$PrimitiveLongConcatingIterator.fetchNext(PrimitiveLongCollections.java:196)\r\n\tat org.neo4j.collection.primitive.PrimitiveLongCollections$PrimitiveLongBaseIterator.hasNext(PrimitiveLongCollections.java:56)\r\n\tat org.neo4j.collection.primitive.PrimitiveLongCollections$14.hasNext(PrimitiveLongCollections.java:740)\r\n\tat org.neo4j.helpers.collection.ResourceClosingIterator.hasNext(ResourceClosingIterator.java:61)\r\n\tat com.graphaware.tx.executor.batch.IndexBug.processBatch(IndexBug.java:90)\r\n```\r\n\r\nIs there any way to iterate over all nodes with label without having to do it in a single transaction? Thanks\r\n\r\nThe test\r\n\r\n```java\r\npublic class IndexBug {\r\n\r\n    private static final int NUMBER_OF_NODES = 132_000;\r\n    private static final int BATCH_SIZE = 1_000;\r\n    private GraphDatabaseService database;\r\n\r\n    @Before\r\n    public void setUp() throws Exception {\r\n        database = new TestGraphDatabaseFactory()\r\n                .newImpermanentDatabaseBuilder()\r\n                .newGraphDatabase();\r\n\r\n        try (Transaction tx = database.beginTx()) {\r\n            for (int i = 0; i < NUMBER_OF_NODES; i++) {\r\n                Node node = database.createNode();\r\n                node.addLabel(DynamicLabel.label(\"FirstLabel\"));\r\n            }\r\n            tx.success();\r\n        }\r\n    }\r\n\r\n    @After\r\n    public void tearDown() throws Exception {\r\n        database.shutdown();\r\n    }\r\n\r\n    @Test\r\n    public void testLabelsCanBeAddedInBatch() {\r\n        Iterator<Node> allNodes;\r\n\r\n        try (Transaction tx = database.beginTx()) {\r\n            allNodes = Iterables.asResourceIterable(database.findNodes(DynamicLabel.label(\"FirstLabel\"))).iterator();\r\n            tx.success();\r\n        }\r\n\r\n        int counter = 0;\r\n        while (processBatch(allNodes)) {\r\n            System.out.println(\"Processed Batch \"+ ++counter);\r\n        }\r\n\r\n        int i = 0;\r\n        try (Transaction tx = database.beginTx()) {\r\n            ResourceIterator<Node> nodes = database.findNodes(DynamicLabel.label(\"SecondLabel\"));\r\n            while (nodes.hasNext()) {\r\n                i++;\r\n                nodes.next();\r\n            }\r\n\r\n            tx.success();\r\n        }\r\n\r\n        assertEquals(NUMBER_OF_NODES, i);\r\n    }\r\n\r\n    private boolean processBatch(Iterator<Node> allNodes) {\r\n        boolean result = true;\r\n\r\n        try (Transaction tx = database.beginTx()) {\r\n            for (int i = 0; i < BATCH_SIZE; i++) {\r\n                if (!allNodes.hasNext()) {\r\n                    result = false;\r\n                    break;\r\n                }\r\n\r\n                Node next = allNodes.next();\r\n                next.addLabel(DynamicLabel.label(\"SecondLabel\"));\r\n            }\r\n            tx.success();\r\n        }\r\n\r\n        return result;\r\n    }\r\n}\r\n```\r\n\r\nTested on Neo4j 2.3.1"},"comment":{"url":"https://api.github.com/repos/neo4j/neo4j/issues/comments/172784886","html_url":"https://github.com/neo4j/neo4j/issues/6087#issuecomment-172784886","issue_url":"https://api.github.com/repos/neo4j/neo4j/issues/6087","id":172784886,"user":{"login":"spacecowboy","id":223655,"avatar_url":"https://avatars.githubusercontent.com/u/223655?v=3","gravatar_id":"","url":"https://api.github.com/users/spacecowboy","html_url":"https://github.com/spacecowboy","followers_url":"https://api.github.com/users/spacecowboy/followers","following_url":"https://api.github.com/users/spacecowboy/following{/other_user}","gists_url":"https://api.github.com/users/spacecowboy/gists{/gist_id}","starred_url":"https://api.github.com/users/spacecowboy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spacecowboy/subscriptions","organizations_url":"https://api.github.com/users/spacecowboy/orgs","repos_url":"https://api.github.com/users/spacecowboy/repos","events_url":"https://api.github.com/users/spacecowboy/events{/privacy}","received_events_url":"https://api.github.com/users/spacecowboy/received_events","type":"User","site_admin":false},"created_at":"2016-01-19T09:15:37Z","updated_at":"2016-01-19T09:15:37Z","body":"@bachmanm Sorry for the delay. Didn't catch your replies.\r\n\r\nDecided to re-run it today and was able to reproduce it without changing `NUMBER_OF_NODES` on latest neo4j-2.3.2-SNAPSHOT and jdk8. \r\n\r\nI have raised this internally."}}