{"action":"closed","issue":{"url":"https://api.github.com/repos/tensorflow/ecosystem/issues/106","repository_url":"https://api.github.com/repos/tensorflow/ecosystem","labels_url":"https://api.github.com/repos/tensorflow/ecosystem/issues/106/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/ecosystem/issues/106/comments","events_url":"https://api.github.com/repos/tensorflow/ecosystem/issues/106/events","html_url":"https://github.com/tensorflow/ecosystem/issues/106","id":370438043,"node_id":"MDU6SXNzdWUzNzA0MzgwNDM=","number":106,"title":"[TFRecords File is too large] 3 times than parquet, 1.3 times than normal textFile","user":{"login":"Zachary4biz","id":22313172,"node_id":"MDQ6VXNlcjIyMzEzMTcy","avatar_url":"https://avatars1.githubusercontent.com/u/22313172?v=4","gravatar_id":"","url":"https://api.github.com/users/Zachary4biz","html_url":"https://github.com/Zachary4biz","followers_url":"https://api.github.com/users/Zachary4biz/followers","following_url":"https://api.github.com/users/Zachary4biz/following{/other_user}","gists_url":"https://api.github.com/users/Zachary4biz/gists{/gist_id}","starred_url":"https://api.github.com/users/Zachary4biz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zachary4biz/subscriptions","organizations_url":"https://api.github.com/users/Zachary4biz/orgs","repos_url":"https://api.github.com/users/Zachary4biz/repos","events_url":"https://api.github.com/users/Zachary4biz/events{/privacy}","received_events_url":"https://api.github.com/users/Zachary4biz/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-10-16T04:47:20Z","updated_at":"2018-10-31T03:38:42Z","closed_at":"2018-10-31T03:38:42Z","author_association":"NONE","body":"**As  a binary file, shouldn't TFRecords-file be much smaller than other types of file?(especially some textFile of RDD)**\r\n\r\nI have created this \"featureDF\", and saved it in 3 different ways.  \r\nparquet: **793.4M**\r\n`featureDF.write.parquet(\"xxx\")`\r\ntextFile: **1.8G**\r\n`featureDF.rdd.saveAsTextFile(\"xxxx\")`\r\ntfrecords: **2.4G**\r\n`featureDF.write.format(\"tfrecords\").options(\"recordType\",\"Example\").save(\"xxx\")`\r\n\r\nand with tfrecords, `options(\"codec\",\"org.apache.hadoop.io.compress.GzipCodec\")` is not working.\r\nso i tried the linux command `gzip -9 featureDF.tfrecord`, compressed it from 2.4G to 515M, cost 5 min. But the `gzip -9 featureDF.parquet` compressed a parquet file from 793.4M to 149M.\r\n\r\nIs there any efficient way to compress this tfrecord files?\r\n(already tried replace DoubleType and LongTpye with StringType,  not useful)\r\n\r\n\r\n"}}
{"actor":{"display_login":"Zachary4biz"}}