{"action":"created","issue":{"url":"https://api.github.com/repos/intel/mkl-dnn/issues/282","repository_url":"https://api.github.com/repos/intel/mkl-dnn","labels_url":"https://api.github.com/repos/intel/mkl-dnn/issues/282/labels{/name}","comments_url":"https://api.github.com/repos/intel/mkl-dnn/issues/282/comments","events_url":"https://api.github.com/repos/intel/mkl-dnn/issues/282/events","html_url":"https://github.com/intel/mkl-dnn/issues/282","id":342323448,"node_id":"MDU6SXNzdWUzNDIzMjM0NDg=","number":282,"title":"Question on performance under limited cpu usage","user":{"login":"t13m","id":5397898,"node_id":"MDQ6VXNlcjUzOTc4OTg=","avatar_url":"https://avatars2.githubusercontent.com/u/5397898?v=4","gravatar_id":"","url":"https://api.github.com/users/t13m","html_url":"https://github.com/t13m","followers_url":"https://api.github.com/users/t13m/followers","following_url":"https://api.github.com/users/t13m/following{/other_user}","gists_url":"https://api.github.com/users/t13m/gists{/gist_id}","starred_url":"https://api.github.com/users/t13m/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/t13m/subscriptions","organizations_url":"https://api.github.com/users/t13m/orgs","repos_url":"https://api.github.com/users/t13m/repos","events_url":"https://api.github.com/users/t13m/events{/privacy}","received_events_url":"https://api.github.com/users/t13m/received_events","type":"User","site_admin":false},"labels":[{"id":886920854,"node_id":"MDU6TGFiZWw4ODY5MjA4NTQ=","url":"https://api.github.com/repos/intel/mkl-dnn/labels/sighting","name":"sighting","color":"f9ea5e","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-07-18T13:15:08Z","updated_at":"2018-07-19T17:23:54Z","closed_at":null,"author_association":"NONE","body":"Hi, dear developers, thank you for your great work on mkl-dnn.\r\n\r\nI run inference of a conv-lstm-dnn network on a cpu-only machine, with tensorflow with and without mkldnn integrated. I found the mkldnn version tensorflow is slower than the other when I use only one cpu core (by setting `OMP_THREADS_NUM=1`, `intra_op_parallelism_threads=1`, `inter_op_parallelism_threads=1`)\r\n\r\nAfter profiling, I have two questions\r\n1. `_MklConv2DWithBias` seems much slower than tensorflow's `Conv2D` on my machine, is there anything wrong of my setting? is there any way to accelerate?\r\n2. I'm not sure, but `MatMul` seems not be replaced with mkldnn version of matmul, is there anything wrong?\r\n\r\n-----------------------------------------------------------------------------\r\n\r\n### Environment\r\nIntel MKL-DNN includes hardware-specific optimizations and may behave \r\ndifferently on depending on the compiler and build environment. Include \r\nthe following information to help reproduce the issue:\r\n* CPU make and model (try `lscpu`):\r\n```\r\nArchitecture:          x86_64\r\nCPU op-mode(s):        32-bit, 64-bit\r\nByte Order:            Little Endian\r\nCPU(s):                32\r\nOn-line CPU(s) list:   0-31\r\nThread(s) per core:    2\r\nCore(s) per socket:    8\r\nSocket(s):             2\r\nNUMA node(s):          2\r\nVendor ID:             GenuineIntel\r\nCPU family:            6\r\nModel:                 45\r\nModel name:            Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz\r\nStepping:              6\r\nCPU MHz:               1219.867\r\nCPU max MHz:           3300.0000\r\nCPU min MHz:           1200.0000\r\nBogoMIPS:              5200.07\r\nVirtualization:        VT-x\r\nL1d cache:             32K\r\nL1i cache:             32K\r\nL2 cache:              256K\r\nL3 cache:              20480K\r\nNUMA node0 CPU(s):     0-7,16-23\r\nNUMA node1 CPU(s):     8-15,24-31\r\nFlags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts\r\n```\r\n\r\n* OS version (`uname -a`)\r\n```\r\nLinux localhost.localdomain 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux\r\n```\r\n* Compiler version (`gcc --version`)\r\n```\r\nUsing built-in specs.\r\nCOLLECT_GCC=gcc\r\nCOLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper\r\nTarget: x86_64-redhat-linux\r\nConfigured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux\r\nThread model: posix\r\ngcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) \r\n```\r\n* MKLROOT value (`echo MKLROOT=$MKLROOT`)\r\n```\r\nMKLROOT=\r\n```\r\n* CMake version (`cmake --version`): 3.11.2\r\n* CMake output log\r\n\r\n### Steps to reproduce\r\n\r\nI run inference of a network use the code below, under both case (with or without mkldnn), and generate json file of timeline for profiling.\r\n\r\nThe code was run with these envvar:\r\n```\r\nKMP_BLOCKTIME=0\r\nKMP_AFFINITY=granularity=fine,verbose,compact,1,0 \r\nOMP_NUM_THREADS=1 \r\nKMP_SETTINGS=1\r\n```\r\n\r\nThe profiling jsons are attached as attachment.\r\n[timeline.json.zip](https://github.com/intel/mkl-dnn/files/2205878/timeline.json.zip)\r\n\r\n```python\r\nfrom __future__ import print_function, division\r\nimport numpy as np\r\nimport tensorflow as tf\r\nimport time\r\nimport sys\r\nimport random\r\n\r\nBATCH_SIZE=1\r\n\r\ndef get_cld_model(feat, scope):\r\n    '''\r\n    config[\"n_lstm_layer\", \"n_lstm_unit\", \"n_dnn_layer\", \"n_dnn_unit\", \"n_output_dim\"]\r\n    :param feature: float32, [batch_size, n_steps, n_tot_dim] (n_tot_dim = n_input * (2 * n_context + 1) )\r\n    :param is_training: bool, []\r\n    :return: float32, [batch_size, n_steps, n_output] logits\r\n    '''\r\n\r\n    #PH_STATE = tf.placeholder_with_default()\r\n\r\n    padded_len = tf.shape(feat)[1]\r\n    with tf.variable_scope('CNN') as scope:\r\n        reshaped_feature = tf.reshape(feat, shape=[-1, 11, 30, 1])\r\n        conv1 = tf.layers.conv2d(reshaped_feature, 96, [11, 7], activation=None)\r\n        conv1 = tf.layers.batch_normalization(conv1, training=False)\r\n        conv1 = tf.nn.relu(conv1)\r\n        conv1 = tf.reshape(conv1, shape=[BATCH_SIZE, -1, 2304]) # 2304 ?\r\n\r\n\r\n    lstm_list = []\r\n    zero_states = []\r\n    state_placeholders = []\r\n    with tf.variable_scope('LSTM') as scope:\r\n        for i_layer in range(2):\r\n            lstm = tf.contrib.rnn.LSTMCell(\r\n                576,\r\n                use_peepholes=True,\r\n                forget_bias=1.0,\r\n                state_is_tuple=True)\r\n            zero_states.append(lstm.zero_state(batch_size=BATCH_SIZE, dtype=tf.float32))\r\n            state_placeholders.append(tf.placeholder_with_default(lstm.zero_state(batch_size=BATCH_SIZE, dtype=tf.float32), [2, 1, 576], name='S{}'.format(i_layer)))\r\n            lstm_list.append(lstm)\r\n\r\n        stacked_lstm = tf.contrib.rnn.MultiRNNCell(\r\n            lstm_list, state_is_tuple=True)\r\n\r\n        state_tensor = tuple([tf.contrib.rnn.LSTMStateTuple(ph_layer[0,:,:], ph_layer[1,:,:]) for ph_layer in state_placeholders])\r\n\r\n        outputs, final_states = tf.nn.dynamic_rnn(\r\n            cell=stacked_lstm,\r\n            inputs=conv1,\r\n            initial_state=state_tensor,\r\n            sequence_length=None,\r\n            dtype=tf.float32,\r\n            time_major=False,\r\n            scope=scope\r\n        )\r\n        final_states = [tf.identity(final_states[i], name=\"FS{}\".format(i)) for i in range(len(final_states))]\r\n\r\n    with tf.variable_scope('DNN') as scope:\r\n        for i_layer in range(1, 1):\r\n            outputs = tf.contrib.layers.fully_connected(\r\n                inputs=outputs,\r\n                num_outputs=1728,\r\n                activation_fn=None,\r\n                biases_initializer=tf.random_normal_initializer(stddev=0.04687),\r\n                scope=scope\r\n            )\r\n            outputs = tf.layers.batch_normalization(outputs, training=False)\r\n            outputs = tf.nn.relu(outputs)\r\n    outputs = tf.contrib.layers.fully_connected(\r\n        inputs=outputs,\r\n        num_outputs=8970,\r\n        activation_fn=None,\r\n        scope=scope\r\n    )\r\n    #outputs = tf.nn.log_softmax(outputs, name='LOGIT')\r\n    outputs = tf.nn.softmax(outputs, name='LOGIT')\r\n    return outputs, final_states, state_placeholders\r\n\r\ndef benchmark_unilstm(n_input, n_output, n_layer, n_hidden, n_iter_warmup, n_iter, n_frame):\r\n    ph_x = tf.placeholder(shape=[None, n_input], dtype=tf.float32)\r\n    var_y, var_states, _ = get_cld_model(ph_x, None)\r\n\r\n    conf = tf.ConfigProto()\r\n    conf.inter_op_parallelism_threads = 1\r\n    conf.intra_op_parallelism_threads = 1\r\n\r\n    sess = tf.Session(config=conf)\r\n    sess.run(tf.global_variables_initializer())\r\n\r\n    print(\"Warming up...\", file=sys.stderr)\r\n    tstart = time.time()\r\n    for i in range(n_iter_warmup):\r\n        n_frame_noise = random.choice([-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5])\r\n        tstart_inner = time.time()\r\n        y, states = sess.run([var_y, var_states], feed_dict={ph_x: np.random.random([n_frame + n_frame_noise, n_input])})\r\n        tend_inner = time.time()\r\n        lapse_inner = tend_inner - tstart_inner\r\n        print(\" - %d-th iter of warming up took %f seconds. NFRAME=%d, RTF=%f\" % (i, lapse_inner, y.shape[1], 100 * lapse_inner / float(y.shape[1])), file=sys.stderr)\r\n    tend = time.time()\r\n    lapse = tend - tstart\r\n    rtf_warmup = 100 * lapse / float(n_frame * n_iter_warmup)\r\n    print(\"Warming up %d iterations in %f seconds. RTF=%f\" % (n_iter_warmup, lapse, rtf_warmup), file=sys.stderr)\r\n\r\n    # profiling\r\n    print(\"Profiling...\", file=sys.stderr)\r\n    run_metadata = tf.RunMetadata()\r\n    y, states = sess.run([var_y, var_states], feed_dict={ph_x: np.random.random([n_frame, n_input])}, options=tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE), run_metadata=run_metadata)\r\n    from tensorflow.python.client import timeline\r\n    trace = timeline.Timeline(step_stats=run_metadata.step_stats)\r\n    with open('timeline-fw.json', 'w') as trace_file:\r\n        trace_file.write(trace.generate_chrome_trace_format())\r\n    print(\"Profiling Done ...\", file=sys.stderr)\r\n\r\n    print(\"Running...\", file=sys.stderr)\r\n    tstart = time.time()\r\n    for i in range(n_iter):\r\n        n_frame_noise = random.choice([-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5])\r\n        tstart_inner = time.time()\r\n        y, states = sess.run([var_y, var_states], feed_dict={ph_x: np.random.random([n_frame + n_frame_noise, n_input])})\r\n        tend_inner = time.time()\r\n        lapse_inner = tend_inner - tstart_inner\r\n        print(\" - %d-th iter took %f seconds. NFRAME=%d, RTF=%f\" % (i, lapse_inner, y.shape[1], 100 * lapse_inner / float(y.shape[1])), file=sys.stderr)\r\n    tend = time.time()\r\n    lapse = tend - tstart\r\n    rtf = 100 * lapse / float(n_frame * n_iter)\r\n    print(\"%d iterations in %f seconds. RTF=%f\" % (n_iter, lapse, rtf), file=sys.stderr)\r\n    print(\"%d, %d, %d, %d, %d, %d, %d, %.4f, %.4f\" % (n_input, n_output, n_layer, n_hidden, n_iter_warmup, n_iter, n_frame, rtf_warmup, rtf))\r\n\r\nif __name__ == \"__main__\":\r\n    n_input, n_output, n_layer, n_hidden, n_iter_warmup, n_iter, n_frame = [330, 8970, 2, 576, 100, 500, 48] #[int(item) for item in sys.argv[1:]]\r\n    benchmark_unilstm(n_input, n_output, n_layer, n_hidden, n_iter_warmup, n_iter, n_frame)\r\n```\r\n\r\n### Actual behavior\r\nDescribe the behavior you see.\r\n\r\n### Expected behavior\r\nDescribe the behavior you expect.\r\n\r\nThanks very much.\r\n"},"comment":{"url":"https://api.github.com/repos/intel/mkl-dnn/issues/comments/406353107","html_url":"https://github.com/intel/mkl-dnn/issues/282#issuecomment-406353107","issue_url":"https://api.github.com/repos/intel/mkl-dnn/issues/282","id":406353107,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNjM1MzEwNw==","user":{"login":"emfomenk","id":15356190,"node_id":"MDQ6VXNlcjE1MzU2MTkw","avatar_url":"https://avatars2.githubusercontent.com/u/15356190?v=4","gravatar_id":"","url":"https://api.github.com/users/emfomenk","html_url":"https://github.com/emfomenk","followers_url":"https://api.github.com/users/emfomenk/followers","following_url":"https://api.github.com/users/emfomenk/following{/other_user}","gists_url":"https://api.github.com/users/emfomenk/gists{/gist_id}","starred_url":"https://api.github.com/users/emfomenk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emfomenk/subscriptions","organizations_url":"https://api.github.com/users/emfomenk/orgs","repos_url":"https://api.github.com/users/emfomenk/repos","events_url":"https://api.github.com/users/emfomenk/events{/privacy}","received_events_url":"https://api.github.com/users/emfomenk/received_events","type":"User","site_admin":false},"created_at":"2018-07-19T17:23:54Z","updated_at":"2018-07-19T17:23:54Z","author_association":"CONTRIBUTOR","body":"@jbobba, isn't it because Intel MKL-DNN integration in TF happens w/o Intel MKL BLAS support?\r\nSince the host is below AVX2 we can't use jitted gemm and rely on Intel MKL."}}
{"actor":{"display_login":"emfomenk"}}