{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-ruby/issues/666","repository_url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-ruby","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-ruby/issues/666/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-ruby/issues/666/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-ruby/issues/666/events","html_url":"https://github.com/GoogleCloudPlatform/gcloud-ruby/issues/666","id":153127974,"number":666,"title":"Blob values created as ndb JsonProperty return as Base64 decoded","user":{"login":"bmclean","id":622145,"avatar_url":"https://avatars.githubusercontent.com/u/622145?v=3","gravatar_id":"","url":"https://api.github.com/users/bmclean","html_url":"https://github.com/bmclean","followers_url":"https://api.github.com/users/bmclean/followers","following_url":"https://api.github.com/users/bmclean/following{/other_user}","gists_url":"https://api.github.com/users/bmclean/gists{/gist_id}","starred_url":"https://api.github.com/users/bmclean/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bmclean/subscriptions","organizations_url":"https://api.github.com/users/bmclean/orgs","repos_url":"https://api.github.com/users/bmclean/repos","events_url":"https://api.github.com/users/bmclean/events{/privacy}","received_events_url":"https://api.github.com/users/bmclean/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-ruby/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-ruby/labels/datastore","name":"datastore","color":"207de5"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-05-04T22:18:30Z","updated_at":"2016-05-04T22:26:21Z","closed_at":null,"body":"We have entities in our datastore that were previously created using the ndb [JsonProperty](https://cloud.google.com/appengine/docs/python/ndb/entity-property-reference). For those entities the Gcloud::Datastore::Proto::Value `blob_value` doesn't seem to be returned as Base64 encoded. For example, adding this debugging to Proto.[from_proto_value](https://github.com/GoogleCloudPlatform/gcloud-ruby/blob/f944f6c4cd8cbff6f2a6432999ac36288ad5a89f/lib/gcloud/datastore/proto.rb#L58):\r\n\r\n    elsif !proto_value.blob_value.nil?\r\n      puts \"*\" * 40\r\n      puts proto_value.blob_value\r\n      return StringIO.new(Base64.decode64(proto_value.blob_value))  \r\n\r\nreturns this:\r\n\r\n    ****************************************\r\n    {\"transaction\": {\"crc\": 2325900754, \"entity_version\": 1225}, ...\r\n\r\nWhich is very strange, as when I look at the value for those entities using the cloud.google.com DataStore Viewer they are encoded. Why do these properties return already decoded?\r\n\r\nShould we test that contents of `blob_value` are in fact base64 encoded before attempting to base64 decode?\r\n\r\nShould we instead raise an exception if `blob_value` isn't Base64 encoded? "},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-ruby/issues/comments/217022356","html_url":"https://github.com/GoogleCloudPlatform/gcloud-ruby/issues/666#issuecomment-217022356","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-ruby/issues/666","id":217022356,"user":{"login":"blowmage","id":730,"avatar_url":"https://avatars.githubusercontent.com/u/730?v=3","gravatar_id":"","url":"https://api.github.com/users/blowmage","html_url":"https://github.com/blowmage","followers_url":"https://api.github.com/users/blowmage/followers","following_url":"https://api.github.com/users/blowmage/following{/other_user}","gists_url":"https://api.github.com/users/blowmage/gists{/gist_id}","starred_url":"https://api.github.com/users/blowmage/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/blowmage/subscriptions","organizations_url":"https://api.github.com/users/blowmage/orgs","repos_url":"https://api.github.com/users/blowmage/repos","events_url":"https://api.github.com/users/blowmage/events{/privacy}","received_events_url":"https://api.github.com/users/blowmage/received_events","type":"User","site_admin":false},"created_at":"2016-05-04T22:26:21Z","updated_at":"2016-05-04T22:26:21Z","body":"@bmclean I've looked into this and 1) this is a problem, and 2) this is cloud-ruby's fault. When sending blob bytes over the wire to Datastore through the JSON API we are supposed to base 64 encode the data. So in gcloud-ruby we are currently base 64 encoding the data. But, the Google API Client that we use to connect to the service is also base 64 encoding the data, so in effect we are double encoding the data with base 64. This is why you cannot view blob bytes already in Datastore, since gcloud-ruby is performing a _double decode_ with base 64.\r\n\r\nThis is a terrible situation, but one that gcloud-ruby can address pretty quickly. Unfortunately, this means that all data stored in Datastore by gcoud-ruby 0.8.1 is base64 encoded, and we don't have any mechanism to correct this. You will have to determine which records are incorrect and base64 decode them once we release a fix."}}