{"action":"created","issue":{"url":"https://api.github.com/repos/cloudfoundry-incubator/bosh-vsphere-cpi-release/issues/5","labels_url":"https://api.github.com/repos/cloudfoundry-incubator/bosh-vsphere-cpi-release/issues/5/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry-incubator/bosh-vsphere-cpi-release/issues/5/comments","events_url":"https://api.github.com/repos/cloudfoundry-incubator/bosh-vsphere-cpi-release/issues/5/events","html_url":"https://github.com/cloudfoundry-incubator/bosh-vsphere-cpi-release/issues/5","id":125999366,"number":5,"title":"Bosh-init deployment to vsphere environment fails due to encoding issues","user":{"login":"kitsirota","id":3009800,"avatar_url":"https://avatars.githubusercontent.com/u/3009800?v=3","gravatar_id":"","url":"https://api.github.com/users/kitsirota","html_url":"https://github.com/kitsirota","followers_url":"https://api.github.com/users/kitsirota/followers","following_url":"https://api.github.com/users/kitsirota/following{/other_user}","gists_url":"https://api.github.com/users/kitsirota/gists{/gist_id}","starred_url":"https://api.github.com/users/kitsirota/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kitsirota/subscriptions","organizations_url":"https://api.github.com/users/kitsirota/orgs","repos_url":"https://api.github.com/users/kitsirota/repos","events_url":"https://api.github.com/users/kitsirota/events{/privacy}","received_events_url":"https://api.github.com/users/kitsirota/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-01-11T17:48:47Z","updated_at":"2016-01-26T16:14:28Z","closed_at":null,"body":"We've been trying to deploy bosh into our vSphere environment for a few weeks without much success.     HUGE THANK YOU goes out to @mjavault for tracking down and documenting this bug.  All of the credit for the content below goes to @mjavault.\r\n\r\nWhen we try to run `bosh-init deploy manifest.yml` against our vSphere deployment, we get the following error:\r\n```\r\nDeployment manifest: '/git/manifest.yml'\r\nDeployment state: '/git/manifest-state.json'\r\n​\r\nStarted validating\r\n  Downloading release 'bosh'... Skipped [Found in local cache] (00:00:00)\r\n  Validating release 'bosh'... Finished (00:00:02)\r\n  Downloading release 'bosh-vsphere-cpi'... Skipped [Found in local cache] (00:00:00)\r\n  Validating release 'bosh-vsphere-cpi'... Finished (00:00:00)\r\n  Validating cpi release... Finished (00:00:00)\r\n  Validating deployment manifest... Finished (00:00:00)\r\n  Downloading stemcell... Finished (00:01:03)\r\n  Validating stemcell... Finished (00:00:03)\r\nFinished validating (00:01:09)\r\n​\r\nStarted installing CPI\r\n  Compiling package 'vsphere_cpi_mkisofs/b3ebe039dae6a312784ece4da34d66053d1dfbba'... Finished (00:02:43)\r\n  Compiling package 'vsphere_cpi_ruby/3ce375f2863799664bff235e2c778a3131f1e981'... Finished (00:01:47)\r\n  Compiling package 'vsphere_cpi/6cce7d152770ee8a2d2309d2278cc83af878757d'... Finished (00:00:52)\r\n  Installing packages... Finished (00:00:00)\r\n  Rendering job templates... Finished (00:00:00)\r\n  Installing job 'vsphere_cpi'... Finished (00:00:00)\r\nFinished installing CPI (00:05:24)\r\n​\r\nStarting registry... Finished (00:00:00)\r\nUploading stemcell 'bosh-vsphere-esxi-ubuntu-trusty-go_agent/3149'... Failed (00:01:56)\r\nStopping registry... Finished (00:00:00)\r\nCleaning up rendered CPI jobs... Finished (00:00:00)\r\n​\r\nCommand 'deploy' failed:\r\n  creating stemcell (bosh-vsphere-esxi-ubuntu-trusty-go_agent 3149):\r\n    Unmarshalling external CPI command output: STDOUT: '', STDERR: 'at depth 1 - 19: self signed certificate in certificate chain\r\nI, [2016-01-07T16:06:31.576680 #26031]  INFO -- : Extracting stemcell to: /tmp/d20160107-26031-f6jcqs\r\nI, [2016-01-07T16:06:36.033533 #26031]  INFO -- : Generated name: sc-f8687c5b-30bf-4f10-99e4-4ad8f6cf9901\r\nD, [2016-01-07T16:06:36.033812 #26031] DEBUG -- : All clusters provided: {\"cluster1\"=>#<VSphereCloud::ClusterConfig:0x007f182922c4d0 @name=\"cluster1\", @config={}>}\r\nat depth 1 - 19: self signed certificate in certificate chain\r\nat depth 1 - 19: self signed certificate in certificate chain\r\nD, [2016-01-07T16:08:04.610612 #26031] DEBUG -- : cluster1 ephemeral disk bound\r\nD, [2016-01-07T16:08:04.610796 #26031] DEBUG -- : Acceptable clusters: [[<Cluster: <[Vim.ClusterComputeResource] domain-c63525> / cluster1>, 5929]]\r\nD, [2016-01-07T16:08:04.610881 #26031] DEBUG -- : Choosing cluster by weighted random\r\nD, [2016-01-07T16:08:04.610955 #26031] DEBUG -- : Selected cluster 'cluster1'\r\nD, [2016-01-07T16:08:04.611021 #26031] DEBUG -- : Looking for a ephemeral datastore in cluster1 with 529MB free space.\r\nD, [2016-01-07T16:08:04.611094 #26031] DEBUG -- : All datastores within cluster cluster1: [\"Somerville 3PAR 2 VM 5 (3137714MB free of 4194048MB capacity)\"]\r\nD, [2016-01-07T16:08:04.611172 #26031] DEBUG -- : Datastores with enough space: [\"Somerville 3PAR 2 VM 5 (3137714MB free of 4194048MB capacity)\"]\r\nI, [2016-01-07T16:08:04.611252 #26031]  INFO -- : Deploying to: <[Vim.ClusterComputeResource] domain-c63525> / <[Vim.Datastore] datastore-188772>\r\nI, [2016-01-07T16:08:05.045048 #26031]  INFO -- : Importing VApp\r\nI, [2016-01-07T16:08:05.141158 #26031]  INFO -- : Waiting for NFC lease to become ready\r\nI, [2016-01-07T16:08:07.168564 #26031]  INFO -- : Uploading\r\nI, [2016-01-07T16:08:07.178566 #26031]  INFO -- : Uploading disk to: https://vmhost/nfc/52b32078-1cfe-c4f3-a2e8-94769d02a483/disk-0.vmdk\r\nat depth 0 - 20: unable to get local issuer certificate\r\nI, [2016-01-07T16:08:22.260389 #26031]  INFO -- : Removing NICs\r\nI, [2016-01-07T16:08:23.315795 #26031]  INFO -- : Taking initial snapshot\r\n/home/user/.bosh_init/installations/7f8327b7-578b-4d35-7a62-2ee15c504e92/packages/vsphere_cpi_ruby/lib/ruby/2.1.0/json/common.rb:223:in `encode': \"\\xC2\" on US-ASCII (Encoding::InvalidByteSequenceError)\r\n\tfrom /home/user/.bosh_init/installations/7f8327b7-578b-4d35-7a62-2ee15c504e92/packages/vsphere_cpi_ruby/lib/ruby/2.1.0/json/common.rb:223:in `generate'\r\n\tfrom /home/user/.bosh_init/installations/7f8327b7-578b-4d35-7a62-2ee15c504e92/packages/vsphere_cpi_ruby/lib/ruby/2.1.0/json/common.rb:223:in `generate'\r\n\tfrom /home/user/.bosh_init/installations/7f8327b7-578b-4d35-7a62-\r\n        from /home/user/.bosh_init/installations/7f8327b7-578b-4d35-7a62-2ee15c504e92/packages/vsphere_cpi/vendor/bundle/ruby/2.1.0/gems/bosh_cpi-1.3093.0/lib/bosh/cpi/cli.rb:114:in `result_response'\r\n        from /home/user/.bosh_init/installations/7f8327b7-578b-4d35-7a62-2ee15c504e92/packages/vsphere_cpi/vendor/bundle/ruby/2.1.0/gems/bosh_cpi-1.3093.0/lib/bosh/cpi/cli.rb:82:in `run'\r\n        from /home/user/.bosh_init/installations/7f8327b7-578b-4d35-7a62-2ee15c504e92/packages/vsphere_cpi/bin/vsphere_cpi:37:in `<main>'\r\n':\r\n      unexpected end of JSON input\r\n```\r\n\r\nThe deploy process is downloading a huge xml file that contains UTF8 characters. Now that file is properly parsed by ruby, but the code also specifies that the full output should be added as a string in a hash object, for logging purposes. Later in the code, that hash object is converted back to json, but the encoder fails because it finds invalid characters, specifically UTF8 characters in a ASCII string.(edited)\r\n\r\ncode extract, `cli.rb`:\r\n```  \r\n  def result_response(result)\r\n    hash = {\r\n      result: result,\r\n      error: nil,\r\n      log: @logs_string_io.string,\r\n    }\r\n    @result_io.print(JSON.dump(hash)); nil\r\n  end\r\n```\r\n\r\nThe problem here is the `log: @logs_string_io.string`, the UTF8 encoding is lost. I doubt the variable is actually very useful, but in doubt, I patched it this way:\r\n`log: @logs_string_io.string.force_encoding(Encoding::UTF_8)`\r\n\r\nI have never coded in ruby in my life, so ruby people would probably come up with a cleaner fix, but this effectively forces the string as UTF8, and the later calls to `generate()` work just fine!  With that file patched, the process goes all the way (I'm still getting some warnings, that I assume are related to the way I hacked my patch into the archives) \r\n\r\nSo, until we come up with a cleaner, official fix, here is a step by step guide on how to patch it yourself.\r\n\r\n```\r\n1. Run `bosh-init deploy manifest.yml` once, and let it fail\r\n2. Go into `~/.bosh_init/installations/{install uuid}/blobs` (the `{installation uuid}` can be found in the error message that you got at step 1)\r\n3. look for the largest file in that folder (there should be three files), and note the name (it should look like a UUID)\r\n4. create a `tmp` folder here, and extract the file: `tar -xvzf {file} -C tmp`\r\n5. edit `tmp/vendor/bundle/ruby/2.1.0/gems/bosh_cpi-1.3093.0/lib/bosh/cpi/cli.rb` (note: if the file does not exist, it's likely you didn't pick the right file out of the three)\r\n6. patch line 112: `log: @logs_string_io.string.force_encoding(Encoding::UTF_8),`, save the file\r\n7. from inside the `tmp` folder: `tar -cvzf ..\\patched.tgz *`\r\n8. rename `patched.tgz` to the original file name from step 3 (you might want to rename the original file first as a backup)\r\n9. generate the sha1: `sha1sum {file}` and write it down\r\n10. edit `~/.bosh_init/installations/{install uuid}/compiled_packages.json`, search for the `BlobID` that is the name of the file from step 3, and update the corresponding `BlobSHA1`\r\n11. that's it. Go back into you working folder, and run `bosh-init deploy manifest.yml` one more time.\r\n```"},"comment":{"url":"https://api.github.com/repos/cloudfoundry-incubator/bosh-vsphere-cpi-release/issues/comments/175096215","html_url":"https://github.com/cloudfoundry-incubator/bosh-vsphere-cpi-release/issues/5#issuecomment-175096215","issue_url":"https://api.github.com/repos/cloudfoundry-incubator/bosh-vsphere-cpi-release/issues/5","id":175096215,"user":{"login":"jhunt","id":20047,"avatar_url":"https://avatars.githubusercontent.com/u/20047?v=3","gravatar_id":"","url":"https://api.github.com/users/jhunt","html_url":"https://github.com/jhunt","followers_url":"https://api.github.com/users/jhunt/followers","following_url":"https://api.github.com/users/jhunt/following{/other_user}","gists_url":"https://api.github.com/users/jhunt/gists{/gist_id}","starred_url":"https://api.github.com/users/jhunt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhunt/subscriptions","organizations_url":"https://api.github.com/users/jhunt/orgs","repos_url":"https://api.github.com/users/jhunt/repos","events_url":"https://api.github.com/users/jhunt/events{/privacy}","received_events_url":"https://api.github.com/users/jhunt/received_events","type":"User","site_admin":false},"created_at":"2016-01-26T16:14:28Z","updated_at":"2016-01-26T16:14:28Z","body":"I went ahead and built a dev version of the vsphere-cpi release, and was able to successfully use that to deploy BOSH via `bosh-init`.\r\n\r\nCan we get a version bump to v15?"}}