{"action":"created","issue":{"url":"https://api.github.com/repos/hadikazemi/Machine-Learning/issues/3","repository_url":"https://api.github.com/repos/hadikazemi/Machine-Learning","labels_url":"https://api.github.com/repos/hadikazemi/Machine-Learning/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/hadikazemi/Machine-Learning/issues/3/comments","events_url":"https://api.github.com/repos/hadikazemi/Machine-Learning/issues/3/events","html_url":"https://github.com/hadikazemi/Machine-Learning/issues/3","id":390017069,"node_id":"MDU6SXNzdWUzOTAwMTcwNjk=","number":3,"title":"Split tfrecords","user":{"login":"sigmoidx","id":19985672,"node_id":"MDQ6VXNlcjE5OTg1Njcy","avatar_url":"https://avatars2.githubusercontent.com/u/19985672?v=4","gravatar_id":"","url":"https://api.github.com/users/sigmoidx","html_url":"https://github.com/sigmoidx","followers_url":"https://api.github.com/users/sigmoidx/followers","following_url":"https://api.github.com/users/sigmoidx/following{/other_user}","gists_url":"https://api.github.com/users/sigmoidx/gists{/gist_id}","starred_url":"https://api.github.com/users/sigmoidx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sigmoidx/subscriptions","organizations_url":"https://api.github.com/users/sigmoidx/orgs","repos_url":"https://api.github.com/users/sigmoidx/repos","events_url":"https://api.github.com/users/sigmoidx/events{/privacy}","received_events_url":"https://api.github.com/users/sigmoidx/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-12-12T00:44:50Z","updated_at":"2018-12-12T01:52:22Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nI found your script works well but created a 32 GB sized tfrecord file (I changed 224 to 299). Is there any official tensforflow way to split it into multiple files with a smaller size each?\r\n\r\nThanks,"},"comment":{"url":"https://api.github.com/repos/hadikazemi/Machine-Learning/issues/comments/446431636","html_url":"https://github.com/hadikazemi/Machine-Learning/issues/3#issuecomment-446431636","issue_url":"https://api.github.com/repos/hadikazemi/Machine-Learning/issues/3","id":446431636,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NjQzMTYzNg==","user":{"login":"sigmoidx","id":19985672,"node_id":"MDQ6VXNlcjE5OTg1Njcy","avatar_url":"https://avatars2.githubusercontent.com/u/19985672?v=4","gravatar_id":"","url":"https://api.github.com/users/sigmoidx","html_url":"https://github.com/sigmoidx","followers_url":"https://api.github.com/users/sigmoidx/followers","following_url":"https://api.github.com/users/sigmoidx/following{/other_user}","gists_url":"https://api.github.com/users/sigmoidx/gists{/gist_id}","starred_url":"https://api.github.com/users/sigmoidx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sigmoidx/subscriptions","organizations_url":"https://api.github.com/users/sigmoidx/orgs","repos_url":"https://api.github.com/users/sigmoidx/repos","events_url":"https://api.github.com/users/sigmoidx/events{/privacy}","received_events_url":"https://api.github.com/users/sigmoidx/received_events","type":"User","site_admin":false},"created_at":"2018-12-12T01:52:22Z","updated_at":"2018-12-12T01:52:22Z","author_association":"NONE","body":"Hi, I found out how to split it....here is the code.\r\n```\r\nimport sys\r\nimport glob\r\nimport threading\r\nimport numpy as np\r\nfrom PIL import Image\r\nimport tensorflow as tf\r\nfrom random import shuffle\r\nfrom datetime import datetime\r\n\r\n\r\ndef _int64_feature(value):\r\n  return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))\r\n\r\ndef _bytes_feature(value):\r\n  return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\r\n\r\ndef image_preprocessing(image_file, height, width):  \r\n\timage = Image.open(image_file)\r\n\timage = image.resize((width, height), Image.ANTIALIAS)        \r\n\tnp_image = np.array(image)\r\n\tnp_image = np_image.astype(float)\r\n \r\n\tnew_image = np.zeros((np_image.shape[0], np_image.shape[1], 3), dtype=float)\r\n\r\n\tif len(np_image.shape) == 2: # 1D images\r\n\t\tfor each_channel in range(3):        \r\n\t\t\tnew_image[:,:,each_channel] = np_image\r\n\telse:  # 3D or 4D images..we only take RGB channels\r\n\t\tfor each_channel in range(3):\r\n\t\t\tnew_image[:,:,each_channel] = np_image[:,:,each_channel]\r\n\t# flushing\r\n\tnp_image = []\r\n\r\n\treturn new_image \r\n\r\ndef process_thread(thread_index, ranges, train_addrs, train_labels, num_shards, name):\r\n\theight = 299\r\n\twidth = 299\r\n\tnum_threads = len(ranges)\r\n\tassert not num_shards % num_threads\r\n\tnum_shards_per_batch = int(num_shards / num_threads)\r\n\r\n\tshard_ranges = np.linspace(ranges[thread_index][0],\r\n\t\t\t\t\t\t\t ranges[thread_index][1],\r\n\t\t\t\t\t\t\t num_shards_per_batch + 1).astype(int)\r\n\tnum_files_in_thread = ranges[thread_index][1] - ranges[thread_index][0]\r\n\r\n\tcounter = 0\r\n\tfor s in range(num_shards_per_batch):\r\n\t\tshard = thread_index * num_shards_per_batch + s\r\n\t\toutput_filename = '%s-%.5d-of-%.5d' % (name, shard, num_shards)\r\n\t\t#train_filename = './tf_records/train.tfrecords' \r\n\t\twriter = tf.python_io.TFRecordWriter(output_filename)\r\n\r\n\t\tshard_counter = 0\r\n\t\tfiles_in_shard = np.arange(shard_ranges[s], shard_ranges[s + 1], dtype=int)\r\n\r\n\t\tfor i in files_in_shard:\r\n\t\t\timg = image_preprocessing(train_addrs[i], height, width)\r\n\t\t\tlabel = train_labels[i]\r\n\r\n\t\t\t# Create a feature\r\n\t\t\tfeature = {'train/label': _int64_feature(label),\r\n\t\t\t\t\t   'train/image': _bytes_feature(tf.compat.as_bytes(img.tostring()))}\r\n\t\t\t# Create an example protocol buffer\r\n\t\t\texample = tf.train.Example(features=tf.train.Features(feature=feature))\r\n\r\n\t\t\t# Serialize to string and write on the file\r\n\t\t\twriter.write(example.SerializeToString())\r\n\t\t\tshard_counter += 1\r\n\t\t\tcounter += 1\r\n\r\n\t\t\tif not counter % 1000:\r\n\t\t\t\tprint('%s [thread %d]: Processed %d of %d images in thread batch.' %\r\n\t\t\t\t(datetime.now(), thread_index, counter, num_files_in_thread))\r\n\t\t\t\tsys.stdout.flush()\r\n\r\n\t\twriter.close()\r\n\t\tprint('%s [thread %d]: Wrote %d images to %s' % (datetime.now(), thread_index, shard_counter, output_filename))\r\n\t\tsys.stdout.flush()\r\n\t\tshard_counter = 0\r\n\tprint('%s [thread %d]: Wrote %d images to %d shards.' % (datetime.now(), thread_index, counter, num_files_in_thread))\r\n\tsys.stdout.flush()\r\n\r\nshuffle_data = True  # shuffle the addresses before saving\r\ncat_dog_train_path = '/media/geraldofrivia/Data/Datasets/pets/train/*.jpg'\r\n# read addresses and labels from the 'train' folder\r\naddrs = glob.glob(cat_dog_train_path)  \r\nlabels = [0 if 'cat' in addr else 1 for addr in addrs]  # 0 = Cat, 1 = Dog\r\n\r\n# to shuffle data\r\nif shuffle_data:\r\n\tc = list(zip(addrs, labels))\r\n\tshuffle(c)\r\n\taddrs, labels = zip(*c)\r\n\r\n# Divide the hata into 60% train, 20% validation, and 20% test\r\ntrain_addrs = addrs[0:int(0.6 * len(addrs))]\r\ntrain_labels = labels[0:int(0.6 * len(labels))]\r\nval_addrs = addrs[int(0.6 * len(addrs)):int(0.8 * len(addrs))]\r\nval_labels = labels[int(0.6 * len(addrs)):int(0.8 * len(addrs))]\r\ntest_addrs = addrs[int(0.8 * len(addrs)):]\r\ntest_labels = labels[int(0.8 * len(labels)):]\r\n\r\n# address to save the TFRecords file\r\n# open the TFRecords file\r\n\r\nnum_shards = 16\r\nnum_threads = 8\r\nspacing = np.linspace(0, len(train_addrs), num_threads + 1).astype(np.int)\r\nranges = []\r\nthreads = []\r\nfor i in range(len(spacing) - 1):\r\n\tranges.append([spacing[i], spacing[i + 1]])\r\n\r\nsave_directory = './tf_records/'\r\nfileName = 'train.tfrecords'\r\nname = save_directory + fileName\r\n\r\ncoord = tf.train.Coordinator()\r\n\r\nthreads = []\r\nfor thread_index in range(len(ranges)):\r\n\targs = (thread_index, ranges, train_addrs, train_labels, num_shards, name)\r\n\tt = threading.Thread(target=process_thread, args=args)\r\n\tt.start()\r\n\tthreads.append(t)\r\n\r\n# Wait for all the threads to terminate.\r\ncoord.join(threads)\r\nprint('%s: Finished writing all %d images in data set.' % (datetime.now(), len(train_addrs)))\r\nsys.stdout.flush()\r\n\r\n\r\n\r\n```"}}
{"actor":{"display_login":"sigmoidx"}}