{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/vault/issues/1864","repository_url":"https://api.github.com/repos/hashicorp/vault","labels_url":"https://api.github.com/repos/hashicorp/vault/issues/1864/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/vault/issues/1864/comments","events_url":"https://api.github.com/repos/hashicorp/vault/issues/1864/events","html_url":"https://github.com/hashicorp/vault/issues/1864","id":175441494,"number":1864,"title":"Migrating from HA to non-HA backend, 500 error","user":{"login":"JeanMertz","id":383250,"avatar_url":"https://avatars.githubusercontent.com/u/383250?v=3","gravatar_id":"","url":"https://api.github.com/users/JeanMertz","html_url":"https://github.com/JeanMertz","followers_url":"https://api.github.com/users/JeanMertz/followers","following_url":"https://api.github.com/users/JeanMertz/following{/other_user}","gists_url":"https://api.github.com/users/JeanMertz/gists{/gist_id}","starred_url":"https://api.github.com/users/JeanMertz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JeanMertz/subscriptions","organizations_url":"https://api.github.com/users/JeanMertz/orgs","repos_url":"https://api.github.com/users/JeanMertz/repos","events_url":"https://api.github.com/users/JeanMertz/events{/privacy}","received_events_url":"https://api.github.com/users/JeanMertz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-09-07T08:23:23Z","updated_at":"2016-09-07T09:35:06Z","closed_at":null,"body":"We're migrating our backend from DynamoDB (HA) to S3 (non-HA).\r\n\r\nI used a simple script like this to make this migration possible:\r\n\r\n```ruby\r\n#!/usr/bin/env ruby\r\nrequire 'json'\r\nrequire 's3'\r\n\r\nfile  = File.read('data/0001.json')\r\ndata  = JSON.parse(file)\r\nitems = data['Items']\r\n  .reject { |i| i['Value']['NULL'] }\r\n  .map { |i| [File.join(i['Path']['S'], i['Key']['S']), i['Value']['S']] }\r\n  .to_h\r\n\r\ns3 = S3::Service.new(access_key_id: ENV['AWS_ACCESS_KEY_ID'], secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'])\r\n\r\nunless s3.buckets.find('vault-xxx')\r\n  bucket = s3.buckets.build('vault-xxx')\r\n  bucket.save(location: :eu)\r\nend\r\n\r\nitems.each do |path, content|\r\n  object = bucket.objects.build(path)\r\n  object.content = content\r\n  object.save\r\n  nil\r\nend\r\n```\r\n\r\nThis was after we used [Dynamodump](https://github.com/bchew/dynamodump) to create a JSON dump of the data on DynamoDB.\r\n\r\nIn the end, all seemed okay, as we now have top level directories `core`, `logical`, and `sys`, with all the relevant directories and files (with content) below that.\r\n\r\nHowever, when running a new Vault instance against this backend, it looks like it can't read the data as expected.\r\n\r\nWhen doing `vault status`, the server responds with:\r\n\r\n```\r\n[ERR] core: failed to decode seal configuration error=invalid character 'e' looking for beginning of value\r\n```\r\n\r\nGiven that I've read both here and on Google Groups that the format in which the data is stored is simple key/value, I'd think that moving the data over to S3 would be straight forward.\r\n\r\nThe only thing I can think of is that migrating from an HA backend, to a non-HA backend requires more work, because it doesn't have any locking mechanism or leader selection?\r\n\r\nIs there anything I can do to solve this issue, or is this simply not possible with Vault?\r\n\r\nI know most other issues get closed with \"migration is not supported by us\" or \"ask Google Groups\", but given the cryptic exception (and returned `500` HTTP status code), I think the devs would have the most knowledge about what could possibly be causing this problem."},"comment":{"url":"https://api.github.com/repos/hashicorp/vault/issues/comments/245227539","html_url":"https://github.com/hashicorp/vault/issues/1864#issuecomment-245227539","issue_url":"https://api.github.com/repos/hashicorp/vault/issues/1864","id":245227539,"user":{"login":"JeanMertz","id":383250,"avatar_url":"https://avatars.githubusercontent.com/u/383250?v=3","gravatar_id":"","url":"https://api.github.com/users/JeanMertz","html_url":"https://github.com/JeanMertz","followers_url":"https://api.github.com/users/JeanMertz/followers","following_url":"https://api.github.com/users/JeanMertz/following{/other_user}","gists_url":"https://api.github.com/users/JeanMertz/gists{/gist_id}","starred_url":"https://api.github.com/users/JeanMertz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JeanMertz/subscriptions","organizations_url":"https://api.github.com/users/JeanMertz/orgs","repos_url":"https://api.github.com/users/JeanMertz/repos","events_url":"https://api.github.com/users/JeanMertz/events{/privacy}","received_events_url":"https://api.github.com/users/JeanMertz/received_events","type":"User","site_admin":false},"created_at":"2016-09-07T09:35:06Z","updated_at":"2016-09-07T09:35:06Z","body":"Next issue seems to be unsealing the Vault.\r\n\r\nWhen using the exact same credentials that work on our current Vault cluster (just tested them again), I can't seem to unseal the new Vault instance that talks to our new S3 backend.\r\n\r\ndebug logs show:\r\n\r\n```\r\n==> Vault server started! Log data will stream in below:\r\n\r\n2016/09/07 09:31:35.479684 [DBG] core: cannot unseal, not enough keys keys=1 threshold=3\r\n2016/09/07 09:31:37.972813 [DBG] core: cannot unseal, not enough keys keys=2 threshold=3\r\n2016/09/07 09:31:41.330066 [DBG] core: cannot unseal, not enough keys keys=1 threshold=3\r\n```\r\n\r\nI enter three different keys, and after the third key, it circles back to being sealed and requiring the first of three keys."}}
{"actor":{"display_login":"JeanMertz"}}