{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-mxnet/issues/9026","repository_url":"https://api.github.com/repos/apache/incubator-mxnet","labels_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/9026/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/9026/comments","events_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/9026/events","html_url":"https://github.com/apache/incubator-mxnet/issues/9026","id":281031659,"node_id":"MDU6SXNzdWUyODEwMzE2NTk=","number":9026,"title":"why is it so slow (MXNET0.12)even with NVIDIA V100 GPU?","user":{"login":"dbsxdbsx","id":17372655,"node_id":"MDQ6VXNlcjE3MzcyNjU1","avatar_url":"https://avatars2.githubusercontent.com/u/17372655?v=4","gravatar_id":"","url":"https://api.github.com/users/dbsxdbsx","html_url":"https://github.com/dbsxdbsx","followers_url":"https://api.github.com/users/dbsxdbsx/followers","following_url":"https://api.github.com/users/dbsxdbsx/following{/other_user}","gists_url":"https://api.github.com/users/dbsxdbsx/gists{/gist_id}","starred_url":"https://api.github.com/users/dbsxdbsx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dbsxdbsx/subscriptions","organizations_url":"https://api.github.com/users/dbsxdbsx/orgs","repos_url":"https://api.github.com/users/dbsxdbsx/repos","events_url":"https://api.github.com/users/dbsxdbsx/events{/privacy}","received_events_url":"https://api.github.com/users/dbsxdbsx/received_events","type":"User","site_admin":false},"labels":[{"id":289646717,"node_id":"MDU6TGFiZWwyODk2NDY3MTc=","url":"https://api.github.com/repos/apache/incubator-mxnet/labels/Application","name":"Application","color":"207de5","default":false},{"id":355414483,"node_id":"MDU6TGFiZWwzNTU0MTQ0ODM=","url":"https://api.github.com/repos/apache/incubator-mxnet/labels/Data-loading","name":"Data-loading","color":"fbca04","default":false},{"id":707546925,"node_id":"MDU6TGFiZWw3MDc1NDY5MjU=","url":"https://api.github.com/repos/apache/incubator-mxnet/labels/Performance","name":"Performance","color":"fc2929","default":false},{"id":206159507,"node_id":"MDU6TGFiZWwyMDYxNTk1MDc=","url":"https://api.github.com/repos/apache/incubator-mxnet/labels/Question","name":"Question","color":"cc317c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-12-11T14:16:32Z","updated_at":"2018-07-18T20:52:03Z","closed_at":null,"author_association":"NONE","body":"I test my py on AWS EC2 P3.2xlarge(GPU:V100), AMI: ami-77eb3a0f, python version : 2.7. The .py is as follow:\r\n```\r\n'''this is a py used to predict capts num ranging from 4 to 10~\r\nTherefore, the DataIter should predict capt examples of different size, even in a batch_size'''\r\n\r\nfont_name = 'segoeuib.ttf'\r\n\r\nimport sys\r\n\r\nsys.path.insert(0, \"../../python\")\r\nsys.path.append('../')\r\nsys.path.append('../../')\r\nimport mxnet as mx\r\nimport numpy as np\r\nimport cv2, random\r\nfrom captcha.image import ImageCaptcha\r\n\r\n\r\nclass OCRBatch(object):\r\n    def __init__(self, data_names, data, label_names, label):\r\n        self.data = data\r\n        self.label = label\r\n        self.data_names = data_names\r\n        self.label_names = label_names\r\n\r\n    @property\r\n    def provide_data(self):\r\n        return [(n, x.shape) for n, x in zip(self.data_names, self.data)]\r\n\r\n    @property\r\n    def provide_label(self):\r\n        return [(n, x.shape) for n, x in zip(self.label_names, self.label)]\r\n\r\n\r\ndef gen_rand(capt_num):\r\n    buf = \"\"\r\n    for i in range(capt_num):\r\n        buf += str(random.randint(0, 9))\r\n    return buf\r\n\r\n\r\ndef get_label(capt_str, capt_max_num):\r\n    a = [int(x) for x in capt_str]\r\n    # for max 10 label, if the capt_num is less than 10, fulfill with -1\r\n    min_pos = len(a)\r\n    for i in range(min_pos, capt_max_num):\r\n        a.append(11)  # -1\r\n    return np.array(a)\r\n\r\n\r\ndef gen_sample(captcha, width, height, capt_num):\r\n    num = gen_rand(capt_num)\r\n    img = captcha.generate(num)\r\n    img = np.fromstring(img.getvalue(), dtype='uint8')\r\n    img = cv2.imdecode(img, cv2.IMREAD_COLOR)\r\n    img = cv2.resize(img, (width, height))\r\n\r\n    img = np.multiply(img, 1 / 255.0)\r\n    img = img.transpose(2, 0, 1)\r\n    return (num, img)\r\n\r\n\r\nclass OCRIter(mx.io.DataIter):\r\n    def __init__(self, count, batch_size, height, width):\r\n        super(OCRIter, self).__init__()\r\n        self.captcha = ImageCaptcha(fonts=[font_name])\r\n\r\n        self.batch_size = batch_size\r\n        self.count = count\r\n        self.height = height\r\n        self.width = width\r\n        self.provide_data = [('data', (batch_size, 3, height, width))]\r\n        self.capt_max_num = 10\r\n        self.provide_label = [('softmax_label', (self.batch_size, self.capt_max_num))]  # ori version\r\n        # self.provide_label = [('softmax1_label', (self.batch_size,)),\r\n        #                       ('softmax2_label', (self.batch_size,)),\r\n        #                       # ('softmax3_label', (self.batch_size,)),\r\n        #                       # ('softmax4_label', (self.batch_size,)),\r\n        #                       ]\r\n\r\n    def __iter__(self):\r\n        for k in range(self.count / self.batch_size):\r\n            data = []\r\n            label = []  # ori version\r\n            # label = [[], [], [], []]\r\n            for i in range(self.batch_size):\r\n                capt_num_for_1_example = np.random.randint(4, 10, size=1)\r\n                capt_num, img = gen_sample(self.captcha, self.width, self.height, capt_num_for_1_example)\r\n                data.append(img)\r\n                label.append(get_label(capt_num, self.capt_max_num))\r\n                # num = [int(x) for x in num]\r\n                # for i in range(4):\r\n                #     label[i].append(num[i])\r\n\r\n            data_all = [mx.nd.array(data)]\r\n            label_all = [mx.nd.array(label)]\r\n            data_names = ['data']\r\n            label_names = ['softmax_label']\r\n\r\n            data_batch = OCRBatch(data_names, data_all, label_names, label_all)  # ori version\r\n            # data_batch = mx.io.DataBatch(data=data_all, label=label_all)\r\n            yield data_batch\r\n\r\n    def reset(self):\r\n        pass\r\n\r\n\r\ndef get_ocrnet():\r\n    data = mx.symbol.Variable('data')\r\n    label = mx.symbol.Variable('softmax_label')\r\n    conv1 = mx.symbol.Convolution(data=data, kernel=(5, 5), num_filter=32)\r\n    pool1 = mx.symbol.Pooling(data=conv1, pool_type=\"max\", kernel=(2, 2), stride=(1, 1))\r\n    relu1 = mx.symbol.Activation(data=pool1, act_type=\"relu\")\r\n\r\n    conv2 = mx.symbol.Convolution(data=relu1, kernel=(5, 5), num_filter=32)\r\n    pool2 = mx.symbol.Pooling(data=conv2, pool_type=\"avg\", kernel=(2, 2), stride=(1, 1))\r\n    relu2 = mx.symbol.Activation(data=pool2, act_type=\"relu\")\r\n\r\n    conv3 = mx.symbol.Convolution(data=relu2, kernel=(3, 3), num_filter=32)\r\n    pool3 = mx.symbol.Pooling(data=conv3, pool_type=\"avg\", kernel=(2, 2), stride=(1, 1))\r\n    relu3 = mx.symbol.Activation(data=pool3, act_type=\"relu\")\r\n\r\n    conv4 = mx.symbol.Convolution(data=relu3, kernel=(3, 3), num_filter=32)\r\n    pool4 = mx.symbol.Pooling(data=conv4, pool_type=\"avg\", kernel=(2, 2), stride=(1, 1))\r\n    relu4 = mx.symbol.Activation(data=pool4, act_type=\"relu\")\r\n\r\n    flatten = mx.symbol.Flatten(data=relu4)\r\n    fc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=256)\r\n    fc21 = mx.symbol.FullyConnected(data=fc1, num_hidden=10)\r\n    fc22 = mx.symbol.FullyConnected(data=fc1, num_hidden=10)\r\n    fc23 = mx.symbol.FullyConnected(data=fc1, num_hidden=10)\r\n    fc24 = mx.symbol.FullyConnected(data=fc1, num_hidden=10)\r\n    fc25 = mx.symbol.FullyConnected(data=fc1, num_hidden=10)\r\n    fc26 = mx.symbol.FullyConnected(data=fc1, num_hidden=10)\r\n    fc27 = mx.symbol.FullyConnected(data=fc1, num_hidden=10)\r\n    fc28 = mx.symbol.FullyConnected(data=fc1, num_hidden=10)\r\n    fc29 = mx.symbol.FullyConnected(data=fc1, num_hidden=10)\r\n    fc210 = mx.symbol.FullyConnected(data=fc1, num_hidden=10)\r\n\r\n    fc2 = mx.symbol.Concat(*[fc21, fc22, fc23, fc24, fc25, fc26, fc27, fc28, fc29, fc210], dim=0)\r\n    label = mx.symbol.transpose(data=label)\r\n    label = mx.symbol.Reshape(data=label, target_shape=(0,))\r\n    return mx.symbol.SoftmaxOutput(data=fc2, label=label, name=\"softmax\")\r\n\r\n\r\ndef Accuracy(label, pred):\r\n    \"\"\"the old version, I just think the pos is calculated wrong~\"\"\"\r\n    label = label.T.reshape((-1,))\r\n    hit = 0\r\n    total = 0\r\n    for i in range(pred.shape[0] / capt_num):\r\n        ok = True\r\n        for j in range(capt_num):\r\n            k = i * capt_num + j\r\n            if np.argmax(pred[k]) != int(label[k]):\r\n                ok = False\r\n                break\r\n        if ok:\r\n            hit += 1\r\n        total += 1\r\n    return 1.0 * hit / total\r\n\r\n\r\ndef Accuracy2(label, pred):\r\n    \"\"\"new  version made by me\"\"\"\r\n    hit = 0\r\n    total = 0\r\n    batch_size = pred.shape[0] / capt_num\r\n    for i in range(batch_size):\r\n        ok = True\r\n        for j in range(capt_num):\r\n            k = j * batch_size + i\r\n            if np.argmax(pred[k]) != int(label[i, j]):\r\n                ok = False\r\n                break\r\n        if ok:\r\n            hit += 1\r\n        total += 1\r\n    return 1.0 * hit / total\r\n\r\n\r\nimport argparse\r\n\r\n\r\ndef parse_args(description):\r\n    parser = argparse.ArgumentParser(description=description)\r\n    parser.add_argument('--batch_size', dest='batch_size', type=int, default=8)\r\n    parser.add_argument('--train_exp_num', dest='train_exp_num', type=int, default=2000)\r\n    parser.add_argument('--epoch_num', dest='epoch_num', type=int, default=50)\r\n    parser.add_argument('--gpu_num', dest='gpu_num', type=int, default=1)\r\n    parser.add_argument('--lr', dest='lr', type=float, default=0.00075)\r\n\r\n    args = parser.parse_args()\r\n    return args\r\n\r\n\r\nif __name__ == '__main__':\r\n    import logging\r\n\r\n    head = '%(asctime)-15s %(message)s'\r\n    logging.basicConfig(level=logging.DEBUG, format=head)\r\n\r\n    args = parse_args('train 4 to 10 capt in 1 net')\r\n    network = get_ocrnet()\r\n    batch_size = args.batch_size  # 50\r\n    train_exp_num = args.train_exp_num  # 10000  # 50000\r\n    test_exp_num = 1000\r\n    epoch_num = args.epoch_num  # 2000\r\n    gpu_num = args.gpu_num\r\n    lr = args.lr\r\n    devs = [mx.gpu(i) for i in range(gpu_num)]\r\n    # model = mx.model.FeedForward(ctx=devs,\r\n    #                              symbol=network,\r\n    #                              num_epoch=epoch_num,\r\n    #                              learning_rate=lr,\r\n    #                              wd=0.00001,\r\n    #                              initializer=mx.init.Xavier(factor_type=\"in\", magnitude=2.34),\r\n    #                              momentum=0.9)\r\n    global capt_num\r\n    capt_num = 10\r\n    data_train = OCRIter(train_exp_num, batch_size, 30, 80)\r\n    data_test = OCRIter(test_exp_num, batch_size, 30, 80)\r\n\r\n    # model.fit(X=data_train,\r\n    #           eval_data=data_test,\r\n    #           eval_metric=[Accuracy, Accuracy2],\r\n    #           batch_end_callback=mx.callback.Speedometer(batch_size, 50),\r\n    #           epoch_end_callback=mx.callback.do_checkpoint(prefix='param', period=2))\r\n\r\n\r\n\r\n\r\n    # new version\r\n    lenet_model = mx.mod.Module(symbol=network, context=devs)\r\n    # train with the same\r\n    lenet_model.fit(data_train,\r\n                    eval_data=data_test,\r\n                    optimizer='sgd',\r\n                    optimizer_params={'learning_rate': lr, 'momentum': 0.9, 'wd': 0.00001},\r\n                    eval_metric=[Accuracy, Accuracy2],\r\n                    initializer=mx.init.Xavier(factor_type=\"in\", magnitude=2.34),\r\n                    batch_end_callback=mx.callback.Speedometer(batch_size, 50),\r\n                    epoch_end_callback=mx.callback.do_checkpoint(prefix='param', period=2),\r\n                    num_epoch=epoch_num)\r\n\r\n\r\n    # model.save(\"cnn-ocr\")\r\n```\r\nOn my own host, win10, mx0.12, gpu:940M,  I got near 110 samples/seconds with default params,  but surprisingly, on p3.2xlarge, I got only 170 samples/seconds. In detail, with `watch -n 1 nvidia-smi`,  I found the volatile GPU utile is always near 0%, up t0 4%.  WHY???   Is that just because I got a custom DataIter?\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/incubator-mxnet/issues/comments/406070421","html_url":"https://github.com/apache/incubator-mxnet/issues/9026#issuecomment-406070421","issue_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/9026","id":406070421,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNjA3MDQyMQ==","user":{"login":"safrooze","id":14987847,"node_id":"MDQ6VXNlcjE0OTg3ODQ3","avatar_url":"https://avatars3.githubusercontent.com/u/14987847?v=4","gravatar_id":"","url":"https://api.github.com/users/safrooze","html_url":"https://github.com/safrooze","followers_url":"https://api.github.com/users/safrooze/followers","following_url":"https://api.github.com/users/safrooze/following{/other_user}","gists_url":"https://api.github.com/users/safrooze/gists{/gist_id}","starred_url":"https://api.github.com/users/safrooze/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/safrooze/subscriptions","organizations_url":"https://api.github.com/users/safrooze/orgs","repos_url":"https://api.github.com/users/safrooze/repos","events_url":"https://api.github.com/users/safrooze/events{/privacy}","received_events_url":"https://api.github.com/users/safrooze/received_events","type":"User","site_admin":false},"created_at":"2018-07-18T20:52:03Z","updated_at":"2018-07-18T20:52:03Z","author_association":"CONTRIBUTOR","body":"@dbsxdbsx It's quite easy to use DataLoader with a custom DataSet object. Your custom DataSet class needs to implement only two functions: `__len__()` and `__getitem_()`. You can then easily use DataLoader with num_workers>1. Here is a dummy example of a custom dataset class that contains 1000 elements, each one the index plus some random noise:\r\n\r\n```\r\nclass MyRandomDataset(object):\r\n    def __getitem__(self, idx):\r\n        return nd.array([idx]) + nd.random.normal()\r\n\r\n    def __len__(self):\r\n        return 1000\r\n```"}}
{"actor":{"display_login":"safrooze"}}