{"action":"created","issue":{"url":"https://api.github.com/repos/marklogic/data-movement/issues/139","repository_url":"https://api.github.com/repos/marklogic/data-movement","labels_url":"https://api.github.com/repos/marklogic/data-movement/issues/139/labels{/name}","comments_url":"https://api.github.com/repos/marklogic/data-movement/issues/139/comments","events_url":"https://api.github.com/repos/marklogic/data-movement/issues/139/events","html_url":"https://github.com/marklogic/data-movement/issues/139","id":176699246,"number":139,"title":"Demonstrate restarting failed jobs","user":{"login":"sammefford","id":298950,"avatar_url":"https://avatars.githubusercontent.com/u/298950?v=3","gravatar_id":"","url":"https://api.github.com/users/sammefford","html_url":"https://github.com/sammefford","followers_url":"https://api.github.com/users/sammefford/followers","following_url":"https://api.github.com/users/sammefford/following{/other_user}","gists_url":"https://api.github.com/users/sammefford/gists{/gist_id}","starred_url":"https://api.github.com/users/sammefford/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sammefford/subscriptions","organizations_url":"https://api.github.com/users/sammefford/orgs","repos_url":"https://api.github.com/users/sammefford/repos","events_url":"https://api.github.com/users/sammefford/events{/privacy}","received_events_url":"https://api.github.com/users/sammefford/received_events","type":"User","site_admin":false},"labels":[{"id":297665158,"url":"https://api.github.com/repos/marklogic/data-movement/labels/fix","name":"fix","color":"fbca04","default":false},{"id":297665129,"url":"https://api.github.com/repos/marklogic/data-movement/labels/major","name":"major","color":"fbca04","default":false},{"id":414534170,"url":"https://api.github.com/repos/marklogic/data-movement/labels/recipe","name":"recipe","color":"f9d0c4","default":false},{"id":297664738,"url":"https://api.github.com/repos/marklogic/data-movement/labels/Task","name":"Task","color":"bfdadc","default":false}],"state":"open","locked":false,"assignee":{"login":"sammefford","id":298950,"avatar_url":"https://avatars.githubusercontent.com/u/298950?v=3","gravatar_id":"","url":"https://api.github.com/users/sammefford","html_url":"https://github.com/sammefford","followers_url":"https://api.github.com/users/sammefford/followers","following_url":"https://api.github.com/users/sammefford/following{/other_user}","gists_url":"https://api.github.com/users/sammefford/gists{/gist_id}","starred_url":"https://api.github.com/users/sammefford/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sammefford/subscriptions","organizations_url":"https://api.github.com/users/sammefford/orgs","repos_url":"https://api.github.com/users/sammefford/repos","events_url":"https://api.github.com/users/sammefford/events{/privacy}","received_events_url":"https://api.github.com/users/sammefford/received_events","type":"User","site_admin":false},"assignees":[{"login":"sammefford","id":298950,"avatar_url":"https://avatars.githubusercontent.com/u/298950?v=3","gravatar_id":"","url":"https://api.github.com/users/sammefford","html_url":"https://github.com/sammefford","followers_url":"https://api.github.com/users/sammefford/followers","following_url":"https://api.github.com/users/sammefford/following{/other_user}","gists_url":"https://api.github.com/users/sammefford/gists{/gist_id}","starred_url":"https://api.github.com/users/sammefford/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sammefford/subscriptions","organizations_url":"https://api.github.com/users/sammefford/orgs","repos_url":"https://api.github.com/users/sammefford/repos","events_url":"https://api.github.com/users/sammefford/events{/privacy}","received_events_url":"https://api.github.com/users/sammefford/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/marklogic/data-movement/milestones/3","html_url":"https://github.com/marklogic/data-movement/milestone/3","labels_url":"https://api.github.com/repos/marklogic/data-movement/milestones/3/labels","id":1454577,"number":3,"title":"data-movement-1.0.1","description":"Corresponds with MarkLogic Server 9.0-1","creator":{"login":"sammefford","id":298950,"avatar_url":"https://avatars.githubusercontent.com/u/298950?v=3","gravatar_id":"","url":"https://api.github.com/users/sammefford","html_url":"https://github.com/sammefford","followers_url":"https://api.github.com/users/sammefford/followers","following_url":"https://api.github.com/users/sammefford/following{/other_user}","gists_url":"https://api.github.com/users/sammefford/gists{/gist_id}","starred_url":"https://api.github.com/users/sammefford/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sammefford/subscriptions","organizations_url":"https://api.github.com/users/sammefford/orgs","repos_url":"https://api.github.com/users/sammefford/repos","events_url":"https://api.github.com/users/sammefford/events{/privacy}","received_events_url":"https://api.github.com/users/sammefford/received_events","type":"User","site_admin":false},"open_issues":31,"closed_issues":2,"state":"open","created_at":"2015-12-10T17:57:55Z","updated_at":"2016-12-20T22:30:43Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2016-09-13T17:18:42Z","updated_at":"2016-12-31T00:19:43Z","closed_at":null,"body":"Test with a scenario for each: WriteHostBatcher, ExportListener, ApplyTransformListener, DeleteListener\n"},"comment":{"url":"https://api.github.com/repos/marklogic/data-movement/issues/comments/269838372","html_url":"https://github.com/marklogic/data-movement/issues/139#issuecomment-269838372","issue_url":"https://api.github.com/repos/marklogic/data-movement/issues/139","id":269838372,"user":{"login":"sammefford","id":298950,"avatar_url":"https://avatars.githubusercontent.com/u/298950?v=3","gravatar_id":"","url":"https://api.github.com/users/sammefford","html_url":"https://github.com/sammefford","followers_url":"https://api.github.com/users/sammefford/followers","following_url":"https://api.github.com/users/sammefford/following{/other_user}","gists_url":"https://api.github.com/users/sammefford/gists{/gist_id}","starred_url":"https://api.github.com/users/sammefford/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sammefford/subscriptions","organizations_url":"https://api.github.com/users/sammefford/orgs","repos_url":"https://api.github.com/users/sammefford/repos","events_url":"https://api.github.com/users/sammefford/events{/privacy}","received_events_url":"https://api.github.com/users/sammefford/received_events","type":"User","site_admin":false},"created_at":"2016-12-31T00:19:43Z","updated_at":"2016-12-31T00:19:43Z","body":"Here's a quick brain-dump of some possible failure scenarios for WriteBatcher:\r\n\r\n1. A mal-formed document or uri is rejected\r\n2. An error in MarkLogic server is rejecting all writes\r\n3. A temporary network error causes some batches to fail\r\n4. The client JVM is stopped unexpectedly or otherwise stops functioning\r\n5. The process feeding documents to WriteBatcher fails\r\n6. In addition to 2, 3, or 5 above, new documents have been added to the source, preventing a simple restart with skipping all docs below lastSuccessBatchNumber * batchSize\r\n7. Document updates are coming in and should overwrite some files written previously, but the source cannot provide a version number or lastModified timestamp for each file.\r\n\r\nRecovery options\r\n\r\n1. Register an onBatchFailureListener to retry writing the documents from the failed batch one at a time.  Log which documents fail and what the error is for each.  Then manually review the log and correct each mal-formed document or uri.\r\n2. Register an onBatchSuccessListener to track in a log file the highest number of successful batch (e.g. lastSuccessBatchNumber).  When restarting the job, read lastSuccessBatchNumber from the log file and skip all documents below that highest batch number--that is, iterate over documents in the same order each time and don't send documents to WriteBatcher.add or addAs if they are below  lastSuccessBatchNumber * batchSize.  Optionally, stop the job after a certain number of subsequent batch failures or listener failures.\r\n3. HostAvailabilityListener will most likely detect this scenario and retry the batch against another host on your cluster or stop the job if none are available.  If HostAvailabilityListener is not handling this scenario, either add your exception to HostAvailabilityListener.withHostUnavailableExceptions or create a new listener that handles the scenario better, perhaps calling batcher.retry as appropriate.\r\n4. Handle the same as (2)\r\n5. Handle the same as (2)\r\n6. Register an onBatchSuccessListener to track in a log file the uris (or other unique id) of each successful document.  When restarting the job, read the successful uris from the log file and skip those documents --that is, iterate over source documents and don't send documents to WriteBatcher.add or addAs if they are in the list of successful documents. \r\n7. Register an onBatchSuccessListener to track in a log file the uris and a computed checksum over the contents of each successful document.  When restarting the job, read the successful uris and checksums from the log file and skip those documents where the checksum matches for the source document and the successfully written document.\r\n\r\nHere's a quick brain-dump of some possible failure scenarios for QueryBatcher:\r\n\r\n10. A document in the batch provided to the listener has been deleted before the listener can read the document\r\n11. An error in MarkLogic server is throwing an error on all queries\r\n12. A temporary network error causes some batches to fail\r\n13. The client JVM is stopped unexpectedly or otherwise stops functioning\r\n\r\nRecovery options:\r\n\r\n10. Configure your QueryBatcher by calling withConsistentSnapshot.  Then if you're using ExportListener or ExportToWriterListener configure those by calling withConsistentSnapshot.  If you have a custom listener, get the snapshot timestamp from QueryEvent.getServerTimestamp().  Use that timestamp to read the batch of documents with DocumentManager.read(long serverTimestamp, String... uris) or similar.  That will get you the documents as they were when the job started.\r\n11. Register an onQuerySuccessListener to track in a log file the uris (or other unique id) of each successful document.  When restarting the job, read the successful uris from the log file and skip those documents --that is, iterate over batches of uris matching the query and don't have the custom listener do anything with documents that are in the list of successful documents. \r\n12. Same solution as (3) above\r\n13. Handle the same as (12)\r\n"}}
{"actor":{"display_login":"sammefford"}}