{"action":"created","issue":{"url":"https://api.github.com/repos/pmonks/alfresco-bulk-import/issues/55","repository_url":"https://api.github.com/repos/pmonks/alfresco-bulk-import","labels_url":"https://api.github.com/repos/pmonks/alfresco-bulk-import/issues/55/labels{/name}","comments_url":"https://api.github.com/repos/pmonks/alfresco-bulk-import/issues/55/comments","events_url":"https://api.github.com/repos/pmonks/alfresco-bulk-import/issues/55/events","html_url":"https://github.com/pmonks/alfresco-bulk-import/issues/55","id":162939795,"number":55,"title":"Bulk import from 'Default' failed - java.util.concurrent.RejectedExecutionException","user":{"login":"bkcsfi","id":6412502,"avatar_url":"https://avatars.githubusercontent.com/u/6412502?v=3","gravatar_id":"","url":"https://api.github.com/users/bkcsfi","html_url":"https://github.com/bkcsfi","followers_url":"https://api.github.com/users/bkcsfi/followers","following_url":"https://api.github.com/users/bkcsfi/following{/other_user}","gists_url":"https://api.github.com/users/bkcsfi/gists{/gist_id}","starred_url":"https://api.github.com/users/bkcsfi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bkcsfi/subscriptions","organizations_url":"https://api.github.com/users/bkcsfi/orgs","repos_url":"https://api.github.com/users/bkcsfi/repos","events_url":"https://api.github.com/users/bkcsfi/events{/privacy}","received_events_url":"https://api.github.com/users/bkcsfi/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pmonks/alfresco-bulk-import/labels/bug","name":"bug","color":"e11d21"}],"state":"open","locked":false,"assignee":{"login":"pmonks","id":54865,"avatar_url":"https://avatars.githubusercontent.com/u/54865?v=3","gravatar_id":"","url":"https://api.github.com/users/pmonks","html_url":"https://github.com/pmonks","followers_url":"https://api.github.com/users/pmonks/followers","following_url":"https://api.github.com/users/pmonks/following{/other_user}","gists_url":"https://api.github.com/users/pmonks/gists{/gist_id}","starred_url":"https://api.github.com/users/pmonks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pmonks/subscriptions","organizations_url":"https://api.github.com/users/pmonks/orgs","repos_url":"https://api.github.com/users/pmonks/repos","events_url":"https://api.github.com/users/pmonks/events{/privacy}","received_events_url":"https://api.github.com/users/pmonks/received_events","type":"User","site_admin":false},"assignees":[{"login":"pmonks","id":54865,"avatar_url":"https://avatars.githubusercontent.com/u/54865?v=3","gravatar_id":"","url":"https://api.github.com/users/pmonks","html_url":"https://github.com/pmonks","followers_url":"https://api.github.com/users/pmonks/followers","following_url":"https://api.github.com/users/pmonks/following{/other_user}","gists_url":"https://api.github.com/users/pmonks/gists{/gist_id}","starred_url":"https://api.github.com/users/pmonks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pmonks/subscriptions","organizations_url":"https://api.github.com/users/pmonks/orgs","repos_url":"https://api.github.com/users/pmonks/repos","events_url":"https://api.github.com/users/pmonks/events{/privacy}","received_events_url":"https://api.github.com/users/pmonks/received_events","type":"User","site_admin":false}],"milestone":null,"comments":16,"created_at":"2016-06-29T14:28:09Z","updated_at":"2016-07-10T21:24:07Z","closed_at":null,"body":"Running bulk import tool 2.0.1 on alfresco community edition 5.1.0 (r127059-b7) with postgres on ubuntu 16.04  (8GB ram, 8 vcpu)\r\n\r\nI am attempting to import 565921 pdf files that had been exported from alfresco Community - v4.2.0 (r63893-b12) using version 0.0.6 of this bulk export tool https://github.com/gsdenys/alfresco-bulk-export\r\n\r\nThe export generated a total of 195614  folders, 565921 pdf files and 772691 properties.xml files (about 40GB). Export results\r\n\r\n```\r\n\r\n\tPerformed Export with the following Parameters :\r\n\t   export folder   : /export\r\n\t   node to export  : workspace://SpacesStore/8ac68ee8-31b2-47ed-8c3f-eac8020a5935\r\n\t   ignore exported : false\r\n\t   export versions : false\r\n\t   bulk import revision scheme: true\r\n\tExport elapsed time: minutes:568 , seconds: 34131\r\n```\r\n\r\nDuring import, approximately 221599 pdf files get imported or skipped (initial import, and also re-importing with replace = false) and then the import process fails with an exception.\r\n\r\nThe import process was run with these options\r\n\r\n```\r\nsource default\r\nsource directory /import/Company Home/User Homes/\r\ntarget space /User Homes\r\nreplace = not checked\r\n\r\n```\r\nsample tree output of root of source directory (/import)\r\n\r\nMost folders have 1 pdf file, a few have 2 pdf files. No folders have more than 3 pdf files\r\n\r\n\r\n```\r\n+-- 8ac68ee8-31b2-47ed-8c3f-eac8020a5935.cache\r\n+-- Company Home\r\n|   +-- User Homes\r\n|       +-- general-user\r\n|       |   +-- 000000\r\n|       |   |   +-- 1308\r\n|       |   |   |   +-- entry\r\n|       |   |   |   |   +-- 12112005200321\r\n|       |   |   |   |   |   +-- 12112005200321_cdc_20130827162012.pdf\r\n|       |   |   |   |   |   +-- 12112005200321_cdc_20130827162012.pdf.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005200321.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005200478\r\n|       |   |   |   |   |   +-- 12112005200478_cdc_20130827155542.pdf\r\n|       |   |   |   |   |   +-- 12112005200478_cdc_20130827155542.pdf.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005200478.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005200537\r\n|       |   |   |   |   |   +-- 12112005200537_cdc_20130826164512.pdf\r\n|       |   |   |   |   |   +-- 12112005200537_cdc_20130826164512.pdf.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005200537.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005226138\r\n|       |   |   |   |   |   +-- 12112005226138_cdc_20130827155609.pdf\r\n|       |   |   |   |   |   +-- 12112005226138_cdc_20130827155609.pdf.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005226138.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005226241\r\n|       |   |   |   |   |   +-- 12112005226241_cdc_20130830082522.pdf\r\n|       |   |   |   |   |   +-- 12112005226241_cdc_20130830082522.pdf.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005226241.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005226285\r\n|       |   |   |   |   |   +-- 12112005226285_cdc_20130827155619.pdf\r\n|       |   |   |   |   |   +-- 12112005226285_cdc_20130827155619.pdf.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005226285.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005226398\r\n|       |   |   |   |   |   +-- 12112005226398_cdc_20130830090013.pdf\r\n|       |   |   |   |   |   +-- 12112005226398_cdc_20130830090013.pdf.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005226398.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005226489\r\n|       |   |   |   |   |   +-- 12112005226489_cdc_20130830154047.pdf\r\n|       |   |   |   |   |   +-- 12112005226489_cdc_20130830154047.pdf.metadata.properties.xml\r\n|       |   |   |   |   +-- 12112005226489.metadata.properties.xml\r\n\r\n```\r\n\r\nThe majority of pdf files were created by scanning documents on a Dell scanner. \r\n\r\nSorry I cannot provide any sample content.\r\n\r\nI've re-run the import process with TRACE enabled, sifted through 40GB catalina.out file for this data\r\n\r\n```\r\n     2016-06-27 00:42:42,559 User:admin INFO  [bulkimport.impl.BatchImporterImpl] [BulkImport-Importer-0011] BULKIMPORT: Skipping '12112005832290-invoice.pdf' as it already exists in the repository and 'replace existing' is false.\r\n     2016-06-27 00:42:42,559 User:admin TRACE [util.transaction.SpringAwareUserTransaction] [BulkImport-Importer-0011] Completing transaction for [UserTransaction]\r\n     2016-06-27 00:42:42,559 User:admin DEBUG [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Before commit TransactionSychronizationImpl[ txnId=fd1fce7f-b675-4d89-8c0f-07343c031713]\r\n     2016-06-27 00:42:42,559 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Before Prepare - level 0\r\n     2016-06-27 00:42:42,559 User:admin DEBUG [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Before Prepare priorities:[4]\r\n     2016-06-27 00:42:42,559 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,560 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,560 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,560 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,560 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,560 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,560 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,560 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,560 User:admin DEBUG [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Prepared\r\n     2016-06-27 00:42:42,560 User:admin DEBUG [mybatis.spring.SqlSessionUtils] [BulkImport-Importer-0011] Transaction synchronization committing SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@4057dea7]\r\n     2016-06-27 00:42:42,560 User:admin DEBUG [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Before completion: TransactionSychronizationImpl[ txnId=fd1fce7f-b675-4d89-8c0f-07343c031713]\r\n     2016-06-27 00:42:42,560 User:admin DEBUG [mybatis.spring.SqlSessionUtils] [BulkImport-Importer-0011] Transaction synchronization deregistering SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@4057dea7]\r\n     2016-06-27 00:42:42,560 User:admin DEBUG [mybatis.spring.SqlSessionUtils] [BulkImport-Importer-0011] Transaction synchronization closing SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@4057dea7]\r\n     2016-06-27 00:42:42,560 User:admin DEBUG [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] After completion (committed): TransactionSychronizationImpl[ txnId=fd1fce7f-b675-4d89-8c0f-07343c031713]\r\n     2016-06-27 00:42:42,560 User:admin DEBUG [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Bound resource:\r\n     2016-06-27 00:42:42,560 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,560 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,560 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,561 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,562 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,562 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,562 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,563 User:admin TRACE [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Fetched resource:\r\n     2016-06-27 00:42:42,563 User:admin DEBUG [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] After Completion: DONE\r\n     2016-06-27 00:42:42,563 User:admin DEBUG [util.transaction.TransactionSupportUtil] [BulkImport-Importer-0011] Unbound txn synch:TransactionSychronizationImpl[ txnId=fd1fce7f-b675-4d89-8c0f-07343c031713]\r\n     2016-06-27 00:42:42,563 User:admin DEBUG [util.transaction.SpringAwareUserTransaction] [BulkImport-Importer-0011] Committed user transaction: UserTransaction[object=org.alfresco.util.transaction.SpringAwareUserTransaction@1fcb8253, status=3]\r\n     2016-06-27 00:42:42,564 User:admin DEBUG [security.authentication.AuthenticationUtil] [BulkImport-Importer-0011] Removing the current security information.\r\n     2016-06-27 00:42:42,667  TRACE [util.cache.AbstractAsynchronouslyRefreshedCache] [BulkImport-Scanner] get() from cache for key  on AbstractAsynchronouslyRefreshedCache [cacheId=compiledModelsCache]\r\n     2016-06-27 00:42:42,667  TRACE [util.cache.AbstractAsynchronouslyRefreshedCache] [BulkImport-Scanner] get() from cache for key  on AbstractAsynchronouslyRefreshedCache [cacheId=compiledModelsCache]\r\n\r\n```\r\n(at least 10,000 more of the following lines)\r\n```\r\n\r\n     2016-06-27 00:45:33,871  TRACE [util.cache.AbstractAsynchronouslyRefreshedCache] [BulkImport-Scanner] get() from cache for key  on AbstractAsynchronouslyRefreshedCache [cacheId=compiledModelsCache]\r\n     2016-06-27 00:45:33,871  TRACE [util.cache.AbstractAsynchronouslyRefreshedCache] [BulkImport-Scanner] get() from cache for key  on AbstractAsynchronouslyRefreshedCache [cacheId=compiledModelsCache]\r\n     2016-06-27 00:45:33,871  TRACE [util.cache.AbstractAsynchronouslyRefreshedCache] [BulkImport-Scanner] get() from cache for key  on AbstractAsynchronouslyRefreshedCache [cacheId=compiledModelsCache]\r\n     2016-06-27 00:45:33,871  TRACE [util.cache.AbstractAsynchronouslyRefreshedCache] [BulkImport-Scanner] get() from cache for key  on AbstractAsynchronouslyRefreshedCache [cacheId=compiledModelsCache]\r\n     2016-06-27 00:45:33,871  TRACE [util.cache.AbstractAsynchronouslyRefreshedCache] [BulkImport-Scanner] get() from cache for key  on AbstractAsynchronouslyRefreshedCache [cacheId=compiledModelsCache]\r\n\r\n```\r\nthen\r\n\r\n```\r\n     2016-06-27 00:45:33,968  ERROR [bulkimport.impl.Scanner] [BulkImport-Scanner] BULKIMPORT: Bulk import from 'Default' failed.\r\n     java.util.concurrent.RejectedExecutionException: Task org.alfresco.extension.bulkimport.impl.Scanner$BatchImportJob@36b0e96c rejected from\r\n     org.alfresco.extension.bulkimport.impl.BulkImportThreadPoolExecutor@4615da6f[Running, pool size = 16, active threads = 16, queued tasks = 100, completed tasks = 269]\r\n            at org.alfresco.extension.bulkimport.impl.Scanner.submitBatch(Scanner.java:370)\r\n            at org.alfresco.extension.bulkimport.impl.Scanner.submitCurrentBatch(Scanner.java:329)\r\n            at org.alfresco.extension.bulkimport.impl.Scanner.submit(Scanner.java:298)\r\n            at org.alfresco.extension.bulkimport.source.fs.FilesystemBulkImportSource.scanDirectory(FilesystemBulkImportSource.java:223)\r\n            at org.alfresco.extension.bulkimport.source.fs.FilesystemBulkImportSource.scanDirectory(FilesystemBulkImportSource.java:245)\r\n            at org.alfresco.extension.bulkimport.source.fs.FilesystemBulkImportSource.scanDirectory(FilesystemBulkImportSource.java:245)\r\n            at org.alfresco.extension.bulkimport.source.fs.FilesystemBulkImportSource.scanDirectory(FilesystemBulkImportSource.java:245)\r\n            at org.alfresco.extension.bulkimport.source.fs.FilesystemBulkImportSource.scanDirectory(FilesystemBulkImportSource.java:245)\r\n            at org.alfresco.extension.bulkimport.source.fs.FilesystemBulkImportSource.scanDirectory(FilesystemBulkImportSource.java:245)\r\n            at org.alfresco.extension.bulkimport.source.fs.FilesystemBulkImportSource.scanFiles(FilesystemBulkImportSource.java:172)\r\n            at org.alfresco.extension.bulkimport.impl.Scanner.run(Scanner.java:188)\r\n\r\n``` \r\nexample export status results (I clicked stop hour after the import process failed)\r\n\r\n```\r\n    Status:    Stopped\r\n    Initiating User:    admin\r\n    Source Name:    Default\r\n    Source directory    /import/Company Home/User Homes\r\n    Target Space:    /Company Home/User Homes\r\n    Import Type:    Streaming\r\n    Dry run:    No\r\n    Batch Weight:    100\r\n    Threads:    0 active of 0 total\r\n    Start Date:    2016-06-27T02:20:19Z\r\n    Scan End Date:    n/a\r\n    End Date:    2016-06-27T09:51:11Z\r\n    Scan Duration:    07h 30m 52s 457.031ms\r\n    Duration:    07h 30m 51s 990.186ms\r\n    Currently Importing:   \r\n    Source (read) Statistics\r\n    Directories scanned:    217557    8.042 / sec\r\n    Files scanned:    1449099    53.566 / sec\r\n\r\n    Target (write) Statistics\r\n    Aspects associated:    1043712    38.582 / sec\r\n    Batches completed:    2226    0.082 / sec\r\n    Batches submitted:    2343    0.087 / sec\r\n    Bytes imported:    2530418309    93539.094 / sec\r\n    Content streamed:    0    0 / sec\r\n    In place content linked:    0    0 / sec\r\n    Metadata properties imported:    2915017    107.756 / sec\r\n    Nodes imported:    222599    8.229 / sec\r\n    Nodes skipped:    222599    8.229 / sec\r\n    Out-of-order retries:    0    0 / sec\r\n    Versions imported:    0    0 / sec\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/pmonks/alfresco-bulk-import/issues/comments/231611765","html_url":"https://github.com/pmonks/alfresco-bulk-import/issues/55#issuecomment-231611765","issue_url":"https://api.github.com/repos/pmonks/alfresco-bulk-import/issues/55","id":231611765,"user":{"login":"pmonks","id":54865,"avatar_url":"https://avatars.githubusercontent.com/u/54865?v=3","gravatar_id":"","url":"https://api.github.com/users/pmonks","html_url":"https://github.com/pmonks","followers_url":"https://api.github.com/users/pmonks/followers","following_url":"https://api.github.com/users/pmonks/following{/other_user}","gists_url":"https://api.github.com/users/pmonks/gists{/gist_id}","starred_url":"https://api.github.com/users/pmonks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pmonks/subscriptions","organizations_url":"https://api.github.com/users/pmonks/orgs","repos_url":"https://api.github.com/users/pmonks/repos","events_url":"https://api.github.com/users/pmonks/events{/privacy}","received_events_url":"https://api.github.com/users/pmonks/received_events","type":"User","site_admin":false},"created_at":"2016-07-10T21:24:07Z","updated_at":"2016-07-10T21:24:07Z","body":"Are you sure that log file was generated with the [second test build](https://github.com/pmonks/alfresco-bulk-import/files/355428/alfresco-bulk-import-2.0.2-SNAPSHOT.zip) I provided installed?\r\n\r\nThe series of exceptions starting at line 186370 in the [log file](https://github.com/pmonks/alfresco-bulk-import/files/355799/bulk-import.log2.zip) look promising, but the line numbers in the stack trace don't correlate with the code in that second build."}}
{"actor":{"display_login":"pmonks"}}