{"action":"created","issue":{"url":"https://api.github.com/repos/candlepin/candlepin/issues/1263","repository_url":"https://api.github.com/repos/candlepin/candlepin","labels_url":"https://api.github.com/repos/candlepin/candlepin/issues/1263/labels{/name}","comments_url":"https://api.github.com/repos/candlepin/candlepin/issues/1263/comments","events_url":"https://api.github.com/repos/candlepin/candlepin/issues/1263/events","html_url":"https://github.com/candlepin/candlepin/pull/1263","id":157715119,"number":1263,"title":"1313551: Async Import/Export Implementation (merge after branching)","user":{"login":"mstead","id":218800,"avatar_url":"https://avatars.githubusercontent.com/u/218800?v=3","gravatar_id":"","url":"https://api.github.com/users/mstead","html_url":"https://github.com/mstead","followers_url":"https://api.github.com/users/mstead/followers","following_url":"https://api.github.com/users/mstead/following{/other_user}","gists_url":"https://api.github.com/users/mstead/gists{/gist_id}","starred_url":"https://api.github.com/users/mstead/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mstead/subscriptions","organizations_url":"https://api.github.com/users/mstead/orgs","repos_url":"https://api.github.com/users/mstead/repos","events_url":"https://api.github.com/users/mstead/events{/privacy}","received_events_url":"https://api.github.com/users/mstead/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/candlepin/candlepin/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/candlepin/candlepin/labels/Needs%20Rebase","name":"Needs Rebase","color":"fbca04"}],"state":"open","locked":false,"assignee":{"login":"awood","id":17975,"avatar_url":"https://avatars.githubusercontent.com/u/17975?v=3","gravatar_id":"","url":"https://api.github.com/users/awood","html_url":"https://github.com/awood","followers_url":"https://api.github.com/users/awood/followers","following_url":"https://api.github.com/users/awood/following{/other_user}","gists_url":"https://api.github.com/users/awood/gists{/gist_id}","starred_url":"https://api.github.com/users/awood/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/awood/subscriptions","organizations_url":"https://api.github.com/users/awood/orgs","repos_url":"https://api.github.com/users/awood/repos","events_url":"https://api.github.com/users/awood/events{/privacy}","received_events_url":"https://api.github.com/users/awood/received_events","type":"User","site_admin":false},"milestone":null,"comments":20,"created_at":"2016-05-31T16:31:12Z","updated_at":"2016-06-17T12:29:43Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/candlepin/candlepin/pulls/1263","html_url":"https://github.com/candlepin/candlepin/pull/1263","diff_url":"https://github.com/candlepin/candlepin/pull/1263.diff","patch_url":"https://github.com/candlepin/candlepin/pull/1263.patch"},"body":"### Description\r\nThis patch provides the means to perform asynchronous\r\nimports and exports. Going forward the old import/export\r\nAPIs are deprecated, but remain functional.\r\n\r\n### New End Points\r\nGET /consumers/:uuid/export/async\r\nGET /consumers/:uuid/export/:export_id\r\nPOST /owners/:key/imports/async\r\n\r\n### Implementation Details\r\nWhen an async export is initiated, the API will start a new\r\nquartz job that will generated the manifest archive for the\r\nspecified consumer. The async job will store the file by\r\nusing the new ManifestFileService so that it can be downloaded\r\nlater. A stored manifest file can only be downloaded once and is\r\ndeleted afterwards. A new manifest will have to be regenerated in\r\norder to download another. This was done in part to minimize disk\r\nusage.\r\n\r\nThe result data of the export job status will contain the details of\r\nwhere the file can be downloaded.\r\n\r\nWhen an async import is initiated, the job status can be monitored\r\nto determine if the job is still running, failed or succeeded. Any\r\nConflictOverrides can be specified by normal means when making the\r\nrequest. The job's result data can be viewed to determine if there\r\nwere any conflicts during the import.\r\n\r\n**ManifestFileService Notes:**\r\nCurrently the default implementation of this service stores manifest\r\nfiles to the DB. Should we ever want to change this, we would have to\r\ncreate a new implementation and load it in candlepin's guice module.\r\n\r\n**NEW: ManifestCleanerJob**\r\nThe old ExportCleaner job has been deprecated and changed to do nothing. It has been replaced by the ManifestCleanerJob. This job does what the old job did, plus also deletes any files in the manifest file service that have reached the configured expiry.\r\n\r\n### HOW TO TEST\r\n**Create a new distributor consumer.**\r\n`$ sudo subscription-manager register --user admin --pass admin --type candlepin --org admin`\r\n\r\n**Add some entitlements to the distributor.**\r\n`$ sudo subscription-manager list --available --pool-only`\r\n`$ sudo subscription-manager attach --pool <pool_id> --pool <pool_id2 ...`\r\n\r\n**Request an async export for the distributor.**\r\n```\r\n$ sudo subscription-manager identity\r\nsystem identity: 6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d\r\nname: bluestar\r\norg name: Admin Owner\r\norg ID: admin\r\n\r\n$ curl -k -u admin:admin https://localhost:8443/candlepin/consumers/6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d/export/async\r\n{\r\n  \"id\" : \"export_cb4b54ed-77f9-4443-af12-d800b4eb0c44\",\r\n  \"state\" : \"CREATED\",\r\n  \"startTime\" : null,\r\n  \"finishTime\" : null,\r\n  \"result\" : null,\r\n  \"principalName\" : \"admin\",\r\n  \"targetType\" : \"consumer\",\r\n  \"targetId\" : \"6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d\",\r\n  \"ownerId\" : \"admin\",\r\n  \"resultData\" : null,\r\n  \"statusPath\" : \"/jobs/export_cb4b54ed-77f9-4443-af12-d800b4eb0c44\",\r\n  \"done\" : false,\r\n  \"group\" : \"async group\",\r\n  \"created\" : \"2016-05-31T15:24:57+0000\",\r\n  \"updated\" : \"2016-05-31T15:24:57+0000\"\r\n}\r\n\r\n$ curl -k -u admin:admin https://localhost:8443/candlepin/jobs/export_cb4b54ed-77f9-4443-af12-d800b4eb0c44?result_data=true\r\n\r\n{\r\n  \"id\" : \"export_cb4b54ed-77f9-4443-af12-d800b4eb0c44\",\r\n  \"state\" : \"FINISHED\",\r\n  \"startTime\" : \"2016-05-31T15:24:57+0000\",\r\n  \"finishTime\" : \"2016-05-31T15:24:57+0000\",\r\n  \"result\" : \"org.candlepin.sync.ExportResult@388db3fe\",\r\n  \"principalName\" : \"admin\",\r\n  \"targetType\" : \"consumer\",\r\n  \"targetId\" : \"6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d\",\r\n  \"ownerId\" : \"admin\",\r\n  \"resultData\" : {\r\n    \"exportedConsumer\" : \"6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d\",\r\n    \"exportId\" : \"402882e7550715fe0155076af3fe2afc\",\r\n    \"href\" : \"/consumers/6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d/export/download?export_id=402882e7550715fe0155076af3fe2afc\"\r\n  },\r\n  \"statusPath\" : \"/jobs/export_cb4b54ed-77f9-4443-af12-d800b4eb0c44\",\r\n  \"done\" : true,\r\n  \"group\" : \"async group\",\r\n  \"created\" : \"2016-05-31T15:24:57+0000\",\r\n  \"updated\" : \"2016-05-31T15:24:57+0000\"\r\n}\r\n```\r\n\r\n**Use the result data of the job to determine how to download the export.**\r\n```\r\n$ wget --no-check-certificate --content-disposition --user admin --password admin https://bluestar:8443/candlepin/consumers/6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d/export/download?export_id=402882e7550715fe0155076af3fe2afc\r\n```\r\nOr via the cpc tool:\r\n```\r\n./cpc download_consumer_export 6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d 402882e7550715fe0155076af3fe2afc ~/Downloads\r\n~/Downloads/6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d-export.zip\r\n```\r\n\r\n**Create a new owner and import the manifest you downloaded.**\r\n```\r\n./cpc create_owner zombies\r\n{\"parentOwner\"=>nil,\r\n \"id\"=>\"402882e755077d350155078fb1e50008\",\r\n \"key\"=>\"zombies\",\r\n \"displayName\"=>\"zombies\",\r\n \"contentPrefix\"=>nil,\r\n \"defaultServiceLevel\"=>nil,\r\n \"upstreamConsumer\"=>nil,\r\n \"logLevel\"=>nil,\r\n \"href\"=>\"/owners/zombies\",\r\n \"created\"=>\"2016-05-31T16:05:05+0000\",\r\n \"updated\"=>\"2016-05-31T16:05:05+0000\"}\r\n\r\n$ curl -k -X POST -F \"application/zip=@/home/mstead/Downloads/6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d-export.zip\" -u admin:admin https://localhost:8443/candlepin/owners/zombies/imports/async\r\n{\r\n  \"id\" : \"import_7b1f34d9-2d33-4c9a-a532-d1be4d223357\",\r\n  \"state\" : \"CREATED\",\r\n  \"startTime\" : null,\r\n  \"finishTime\" : null,\r\n  \"result\" : null,\r\n  \"principalName\" : \"admin\",\r\n  \"targetType\" : \"owner\",\r\n  \"targetId\" : \"zombies\",\r\n  \"ownerId\" : \"zombies\",\r\n  \"resultData\" : null,\r\n  \"statusPath\" : \"/jobs/import_7b1f34d9-2d33-4c9a-a532-d1be4d223357\",\r\n  \"done\" : false,\r\n  \"group\" : \"async group\",\r\n  \"created\" : \"2016-05-31T16:13:09+0000\",\r\n  \"updated\" : \"2016-05-31T16:13:09+0000\"\r\n}\r\n```\r\n\r\n**Check the job status to see when it has completed.**\r\n```\r\n$ curl -k -u admin:admin https://localhost:8443/candlepin/jobs/import_7b1f34d9-2d33-4c9a-a532-d1be4d223357?result_data=true\r\n{\r\n  \"id\" : \"import_7b1f34d9-2d33-4c9a-a532-d1be4d223357\",\r\n  \"state\" : \"FINISHED\",\r\n  \"startTime\" : \"2016-05-31T16:13:10+0000\",\r\n  \"finishTime\" : \"2016-05-31T16:13:10+0000\",\r\n  \"result\" : \"org.candlepin.model.ImportRecord@692c21e3\",\r\n  \"principalName\" : \"admin\",\r\n  \"targetType\" : \"owner\",\r\n  \"targetId\" : \"zombies\",\r\n  \"ownerId\" : \"zombies\",\r\n  \"resultData\" : {\r\n    \"id\" : \"402882e755077d350155079719690080\",\r\n    \"status\" : \"SUCCESS\",\r\n    \"statusMessage\" : \"zombies file imported successfully.\",\r\n    \"fileName\" : \"6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d-export.zip\",\r\n    \"generatedBy\" : \"admin\",\r\n    \"generatedDate\" : \"2016-05-31T16:09:29+0000\",\r\n    \"upstreamConsumer\" : {\r\n      \"id\" : \"402882e755077d350155079719690081\",\r\n      \"uuid\" : \"6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d\",\r\n      \"name\" : \"bluestar\",\r\n      \"type\" : {\r\n        \"id\" : \"1003\",\r\n        \"label\" : \"candlepin\",\r\n        \"manifest\" : true\r\n      },\r\n      \"ownerId\" : \"402882e755077d350155078fb1e50008\",\r\n      \"webUrl\" : \"localhost:8443/candlepin\",\r\n      \"apiUrl\" : \"localhost:8443/candlepin\",\r\n      \"created\" : \"2016-05-31T16:13:10+0000\",\r\n      \"updated\" : \"2016-05-31T16:13:10+0000\"\r\n    },\r\n    \"created\" : \"2016-05-31T16:13:10+0000\",\r\n    \"updated\" : \"2016-05-31T16:13:10+0000\"\r\n  },\r\n  \"statusPath\" : \"/jobs/import_7b1f34d9-2d33-4c9a-a532-d1be4d223357\",\r\n  \"done\" : true,\r\n  \"group\" : \"async group\",\r\n  \"created\" : \"2016-05-31T16:13:09+0000\",\r\n  \"updated\" : \"2016-05-31T16:13:10+0000\"\r\n}\r\n```\r\n\r\n**Register a system to the new org and verify that the subs from the manifest are available.**\r\n```\r\n$ sudo subscription-manager clean\r\n$ sudo subscription-manager register --user admin --password admin --org zombies\r\n$ sudo list --available\r\n```\r\n\r\n**Verify that conflict overrides can be applied for async imports.**\r\n```\r\n# Import the same manifest again\r\n$ curl -k -X POST -F \"application/zip=@/home/mstead/Downloads/6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d-export.zip\" -u admin:admin https://localhost:8443/candlepin/owners/zombies/imports/async\r\n{\r\n  \"id\" : \"import_a6dcaaa1-06d1-411b-92d0-559a91abc931\",\r\n  \"state\" : \"CREATED\",\r\n  \"startTime\" : null,\r\n  \"finishTime\" : null,\r\n  \"result\" : null,\r\n  \"principalName\" : \"admin\",\r\n  \"targetType\" : \"owner\",\r\n  \"targetId\" : \"zombies\",\r\n  \"ownerId\" : \"zombies\",\r\n  \"resultData\" : null,\r\n  \"statusPath\" : \"/jobs/import_a6dcaaa1-06d1-411b-92d0-559a91abc931\",\r\n  \"done\" : false,\r\n  \"group\" : \"async group\",\r\n  \"created\" : \"2016-05-31T16:20:43+0000\",\r\n  \"updated\" : \"2016-05-31T16:20:43+0000\"\r\n}\r\n\r\n$ curl -k -u admin:admin https://localhost:8443/candlepin/jobs/import_a6dcaaa1-06d1-411b-92d0-559a91abc931?result_data=true\r\n{\r\n  \"id\" : \"import_a6dcaaa1-06d1-411b-92d0-559a91abc931\",\r\n  \"state\" : \"FAILED\",\r\n  \"startTime\" : \"2016-05-31T16:20:43+0000\",\r\n  \"finishTime\" : null,\r\n  \"result\" : \"Import is the same as existing data\",\r\n  \"principalName\" : \"admin\",\r\n  \"targetType\" : \"owner\",\r\n  \"targetId\" : \"zombies\",\r\n  \"ownerId\" : \"zombies\",\r\n  \"resultData\" : {\r\n    \"displayMessage\" : \"Import is the same as existing data\",\r\n    \"requestUuid\" : \"import_a6dcaaa1-06d1-411b-92d0-559a91abc931\",\r\n    \"conflicts\" : [ \"MANIFEST_SAME\" ]\r\n  },\r\n  \"statusPath\" : \"/jobs/import_a6dcaaa1-06d1-411b-92d0-559a91abc931\",\r\n  \"done\" : true,\r\n  \"group\" : \"async group\",\r\n  \"created\" : \"2016-05-31T16:20:43+0000\",\r\n  \"updated\" : \"2016-05-31T16:20:43+0000\"\r\n}\r\n\r\n# Try again with an override\r\n$ curl -k u admin:admin https://localhost:8443/candlepin/jobs/import_ae6e2cd3-0822-4972-9826-a2187b2c4fec?result_data=true\r\n{\r\n  \"id\" : \"import_ae6e2cd3-0822-4972-9826-a2187b2c4fec\",\r\n  \"state\" : \"FINISHED\",\r\n  \"startTime\" : \"2016-05-31T16:23:25+0000\",\r\n  \"finishTime\" : \"2016-05-31T16:23:25+0000\",\r\n  \"result\" : \"org.candlepin.model.ImportRecord@7a588cc2\",\r\n  \"principalName\" : \"admin\",\r\n  \"targetType\" : \"owner\",\r\n  \"targetId\" : \"zombies\",\r\n  \"ownerId\" : \"zombies\",\r\n  \"resultData\" : {\r\n    \"id\" : \"402882e755077d35015507a07bba00ab\",\r\n    \"status\" : \"SUCCESS\",\r\n    \"statusMessage\" : \"zombies file imported forcibly.\",\r\n    \"fileName\" : \"6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d-export.zip\",\r\n    \"generatedBy\" : \"admin\",\r\n    \"generatedDate\" : \"2016-05-31T16:09:29+0000\",\r\n    \"upstreamConsumer\" : {\r\n      \"id\" : \"402882e755077d35015507a07bba00ac\",\r\n      \"uuid\" : \"6970e6eb-2e3f-4adb-b84a-68cfefc7ae8d\",\r\n      \"name\" : \"bluestar\",\r\n      \"type\" : {\r\n        \"id\" : \"1003\",\r\n        \"label\" : \"candlepin\",\r\n        \"manifest\" : true\r\n      },\r\n      \"ownerId\" : \"402882e755077d350155078fb1e50008\",\r\n      \"webUrl\" : \"localhost:8443/candlepin\",\r\n      \"apiUrl\" : \"localhost:8443/candlepin\",\r\n      \"created\" : \"2016-05-31T16:23:25+0000\",\r\n      \"updated\" : \"2016-05-31T16:23:25+0000\"\r\n    },\r\n    \"created\" : \"2016-05-31T16:23:25+0000\",\r\n    \"updated\" : \"2016-05-31T16:23:25+0000\"\r\n  },\r\n  \"statusPath\" : \"/jobs/import_ae6e2cd3-0822-4972-9826-a2187b2c4fec\",\r\n  \"done\" : true,\r\n  \"group\" : \"async group\",\r\n  \"created\" : \"2016-05-31T16:23:24+0000\",\r\n  \"updated\" : \"2016-05-31T16:23:25+0000\"\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/candlepin/candlepin/issues/comments/226756509","html_url":"https://github.com/candlepin/candlepin/pull/1263#issuecomment-226756509","issue_url":"https://api.github.com/repos/candlepin/candlepin/issues/1263","id":226756509,"user":{"login":"mstead","id":218800,"avatar_url":"https://avatars.githubusercontent.com/u/218800?v=3","gravatar_id":"","url":"https://api.github.com/users/mstead","html_url":"https://github.com/mstead","followers_url":"https://api.github.com/users/mstead/followers","following_url":"https://api.github.com/users/mstead/following{/other_user}","gists_url":"https://api.github.com/users/mstead/gists{/gist_id}","starred_url":"https://api.github.com/users/mstead/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mstead/subscriptions","organizations_url":"https://api.github.com/users/mstead/orgs","repos_url":"https://api.github.com/users/mstead/repos","events_url":"https://api.github.com/users/mstead/events{/privacy}","received_events_url":"https://api.github.com/users/mstead/received_events","type":"User","site_admin":false},"created_at":"2016-06-17T12:29:43Z","updated_at":"2016-06-17T12:29:43Z","body":"> @mstead perhaps Im missing some information. I meant to recommend that we remove ExportCleaner job class. is there a reason why we cant do that?\r\n\r\n@vritant DONE. Please review."}}
{"actor":{"display_login":"mstead"}}