{"action":"created","issue":{"url":"https://api.github.com/repos/clMathLibraries/clFFT/issues/139","repository_url":"https://api.github.com/repos/clMathLibraries/clFFT","labels_url":"https://api.github.com/repos/clMathLibraries/clFFT/issues/139/labels{/name}","comments_url":"https://api.github.com/repos/clMathLibraries/clFFT/issues/139/comments","events_url":"https://api.github.com/repos/clMathLibraries/clFFT/issues/139/events","html_url":"https://github.com/clMathLibraries/clFFT/issues/139","id":144798206,"number":139,"title":"Performance drop for certain Y-strides in 2D and 3D real to hermitian interleaved transforms","user":{"login":"MichalKuderjavy","id":18183134,"avatar_url":"https://avatars.githubusercontent.com/u/18183134?v=3","gravatar_id":"","url":"https://api.github.com/users/MichalKuderjavy","html_url":"https://github.com/MichalKuderjavy","followers_url":"https://api.github.com/users/MichalKuderjavy/followers","following_url":"https://api.github.com/users/MichalKuderjavy/following{/other_user}","gists_url":"https://api.github.com/users/MichalKuderjavy/gists{/gist_id}","starred_url":"https://api.github.com/users/MichalKuderjavy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MichalKuderjavy/subscriptions","organizations_url":"https://api.github.com/users/MichalKuderjavy/orgs","repos_url":"https://api.github.com/users/MichalKuderjavy/repos","events_url":"https://api.github.com/users/MichalKuderjavy/events{/privacy}","received_events_url":"https://api.github.com/users/MichalKuderjavy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-03-31T05:59:35Z","updated_at":"2016-03-31T17:39:52Z","closed_at":null,"body":"Hi,\r\n\r\nI observed a significance performance drop for certain Y-strides in 2D and 3D real to hermitian interleaved transforms. I'm using the clFFT-client application for testing.\r\nThis is my device:\r\n\r\n```\r\nOpenCL platform [ 0 ]:\r\n    CL_PLATFORM_PROFILE:     FULL_PROFILE\r\n    CL_PLATFORM_VERSION:     OpenCL 2.0 AMD-APP (2004.6)\r\n    CL_PLATFORM_NAME:        AMD Accelerated Parallel Processing\r\n    CL_PLATFORM_VENDOR:      Advanced Micro Devices, Inc.\r\n    CL_PLATFORM_EXTENSIONS:  cl_khr_icd cl_khr_d3d10_sharing cl_khr_d3d11_sharin\r\ng cl_khr_dx9_media_sharing cl_amd_event_callback cl_amd_offline_devices\r\n\r\nOpenCL platform [ 0 ], device [ 0 ]:\r\n    CL_DEVICE_NAME:                      Hawaii\r\n    CL_DEVICE_VERSION:                   OpenCL 2.0 AMD-APP (2004.6)\r\n    CL_DRIVER_VERSION:                   2004.6 (VM)\r\n    CL_DEVICE_TYPE:                      GPU\r\n    CL_DEVICE_MAX_CLOCK_FREQUENCY:       1025\r\n    CL_DEVICE_ADDRESS_BITS:              64\r\n    CL_DEVICE_AVAILABLE:                 TRUE\r\n    CL_DEVICE_COMPILER_AVAILABLE:        TRUE\r\n    CL_DEVICE_OPENCL_C_VERSION:          OpenCL C 2.0\r\n    CL_DEVICE_MAX_WORK_GROUP_SIZE:       256\r\n    CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS:  3\r\n                         Dimension[ 0 ]  256\r\n                         Dimension[ 1 ]  256\r\n                         Dimension[ 2 ]  256\r\n    CL_DEVICE_HOST_UNIFIED_MEMORY:       FALSE\r\n    CL_DEVICE_MAX_CONSTANT_BUFFER_SIZE:  65536 ( 64 KB )\r\n    CL_DEVICE_LOCAL_MEM_SIZE:            32768 ( 32 KB )\r\n    CL_DEVICE_GLOBAL_MEM_SIZE:           8589934592 ( 8192 MB )\r\n    CL_DEVICE_MAX_MEM_ALLOC_SIZE:        4244635648 ( 4048 MB )\r\n    CL_DEVICE_EXTENSIONS:                cl_khr_fp64 cl_amd_fp64 cl_khr_global_i\r\nnt32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_a\r\ntomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int6\r\n4_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_g\r\nl_sharing cl_khr_gl_depth_images cl_amd_device_attribute_query cl_amd_vec3 cl_am\r\nd_printf cl_amd_media_ops cl_amd_media_ops2 cl_amd_popcnt cl_khr_d3d10_sharing c\r\nl_khr_d3d11_sharing cl_khr_dx9_media_sharing cl_khr_image2d_from_buffer cl_khr_s\r\npir cl_khr_subgroups cl_khr_gl_event cl_khr_depth_images cl_khr_mipmap_image cl_\r\nkhr_mipmap_image_writes\r\n```\r\n\r\n\r\nWhen I set the Y-stride to 542 (which is also the stride that is chosen automatically if I don't specify any) I get this performance:\r\n\r\n```\r\nclFFT-client -x 540 -y 540 -z 8 -p 100 --inLayout 5 --outLayou\r\nt 3 --isY 542 --osY 271 -g\r\n\r\nExecution wall time: 1.54014 ms\r\nExecution gflops: 80.102\r\n\r\n========================StdDev ( 2 )========================\r\nclFFT[ 0 ]: Pruning 1 samples out of 100\r\nclFFT[ 1 ]: Pruning 1 samples out of 100\r\nclFFT[ 2 ]: Pruning 1 samples out of 100\r\nclFFT[ 3 ]: Pruning 1 samples out of 100\r\nclFFT[ 4 ]: Pruning 0 samples out of 100\r\nclFFT[ 5 ]: Pruning 1 samples out of 100\r\nclFFT[ 6 ]: Pruning 2 samples out of 100\r\nclFFT[ 7 ]: Pruning 1 samples out of 100\r\nclFFT[ 8 ]: Pruning 0 samples out of 100\r\n\r\n===========================clFFT============================\r\n        Handle:                   3\r\n        Kernel:    0000000003FF6260\r\n     OutEvents:    00000000053AF4A0\r\n        Length:         (540,540,8)\r\n     Placeness:             InPlace\r\n    Input Dist:             2341440\r\n   Output Dist:             1170720\r\n  Input Stride:      (1,542,292680)\r\n Output Stride:      (1,271,146340)\r\n   Global Work:             (38880)\r\n    Local Work:                (54)\r\n        Gflops:                        454.02\r\n     Time (ns):                                      233,188\r\n\r\n        Handle:                   4\r\n        Kernel:    0000000003FF6800\r\n     OutEvents:    00000000053AF320\r\n        Length:         (271,540,8)\r\n     Placeness:          OutOfPlace\r\n    Input Dist:             1170720\r\n   Output Dist:             1170720\r\n  Input Stride:      (1,271,146340)\r\n Output Stride:      (1,540,146340)\r\n   Global Work:           (144,640)\r\n    Local Work:             (16,16)\r\n        Gflops:                       976.024\r\n     Time (ns):                                      102,909\r\n\r\n        Handle:                   5\r\n        Kernel:    0000000003FF5FC0\r\n     OutEvents:    00000000053B03A0\r\n        Length:         (540,271,8)\r\n     Placeness:             InPlace\r\n    Input Dist:             1170720\r\n   Output Dist:             1170720\r\n  Input Stride:      (1,540,146340)\r\n Output Stride:      (1,540,146340)\r\n   Global Work:             (39042)\r\n    Local Work:                (54)\r\n        Gflops:                       273.336\r\n     Time (ns):                                      194,383\r\n\r\n        Handle:                   6\r\n        Kernel:    000000000263E9B0\r\n     OutEvents:    00000000053AFC20\r\n        Length:         (540,271,8)\r\n     Placeness:          OutOfPlace\r\n    Input Dist:             1170720\r\n   Output Dist:             1170720\r\n  Input Stride:      (1,540,146340)\r\n Output Stride:      (1,271,146340)\r\n   Global Work:           (144,640)\r\n    Local Work:             (16,16)\r\n        Gflops:                       1006.24\r\n     Time (ns):                                       99,818\r\n\r\n        Handle:                   2\r\n Child Handles:           (3,4,5,6)\r\n        Length:         (540,540,8)\r\n     Placeness:             InPlace\r\n    Input Dist:             2341440\r\n   Output Dist:             1170720\r\n  Input Stride:      (1,542,292680)\r\n Output Stride:      (1,271,146340)\r\n        Gflops:                       335.942\r\n     Time (ns):                                      630,299\r\n\r\n        Handle:                   7\r\n        Kernel:    00000000051D42F0\r\n     OutEvents:    00000000053AF7A0\r\n        Length:          (146340,8)\r\n     Placeness:          OutOfPlace\r\n    Input Dist:             1170720\r\n   Output Dist:             1170720\r\n  Input Stride:          (1,146340)\r\n Output Stride:               (1,8)\r\n   Global Work:          (16,36592)\r\n    Local Work:             (16,16)\r\n        Gflops:                       290.197\r\n     Time (ns):                                      406,629\r\n\r\n        Handle:                   8\r\n        Kernel:    00000000051D3750\r\n     OutEvents:    00000000053AFAA0\r\n        Length:          (8,146340)\r\n     Placeness:             InPlace\r\n    Input Dist:             1170720\r\n   Output Dist:             1170720\r\n  Input Stride:               (1,8)\r\n Output Stride:               (1,8)\r\n   Global Work:            (292736)\r\n    Local Work:                (64)\r\n        Gflops:                       278.934\r\n     Time (ns):                                       62,956\r\n\r\n        Handle:                   9\r\n        Kernel:    00000000051D34B0\r\n     OutEvents:    00000000053AEEA0\r\n        Length:          (8,146340)\r\n     Placeness:          OutOfPlace\r\n    Input Dist:             1170720\r\n   Output Dist:             1170720\r\n  Input Stride:               (1,8)\r\n Output Stride:          (1,146340)\r\n   Global Work:          (16,36592)\r\n    Local Work:             (16,16)\r\n        Gflops:                       282.026\r\n     Time (ns):                                      418,409\r\n\r\n        Handle:                   1\r\n Child Handles:           (2,7,9,8)\r\n        Length:         (540,540,8)\r\n     Placeness:             InPlace\r\n    Input Dist:             2341440\r\n   Output Dist:             1170720\r\n  Input Stride:      (1,542,292680)\r\n Output Stride:      (1,271,146340)\r\n        Gflops:                       162.509\r\n     Time (ns):                                    1,518,295\r\n```\r\n\r\n\r\nWhen I change the stride to 544 I get this performance (3x faster):\r\n\r\n```\r\nclFFT-client -x 540 -y 540 -z 8 -p 100 --inLayout 5 --outLayou\r\nt 3 --isY 544 --osY 272 -g\r\n\r\nExecution wall time: 0.479108 ms\r\nExecution gflops: 257.495\r\n\r\n========================StdDev ( 2 )========================\r\nclFFT[ 0 ]: Pruning 5 samples out of 100\r\nclFFT[ 1 ]: Pruning 5 samples out of 100\r\nclFFT[ 2 ]: Pruning 0 samples out of 100\r\nclFFT[ 3 ]: Pruning 1 samples out of 100\r\nclFFT[ 4 ]: Pruning 0 samples out of 100\r\n\r\n===========================clFFT============================\r\n        Handle:                   3\r\n        Kernel:    0000000004206260\r\n     OutEvents:    0000000005380890\r\n        Length:         (540,540,8)\r\n     Placeness:             InPlace\r\n    Input Dist:             2350080\r\n   Output Dist:             1175040\r\n  Input Stride:      (1,544,293760)\r\n Output Stride:      (1,272,146880)\r\n   Global Work:             (38880)\r\n    Local Work:                (54)\r\n        Gflops:                       569.125\r\n     Time (ns):                                      186,025\r\n\r\n        Handle:                   4\r\n        Kernel:    0000000004206DA0\r\n     OutEvents:    0000000005381310\r\n        Length:         (540,271,8)\r\n     Placeness:             InPlace\r\n    Input Dist:             1175040\r\n   Output Dist:             1175040\r\n  Input Stride:      (272,1,146880)\r\n Output Stride:      (272,1,146880)\r\n   Global Work:             (39042)\r\n    Local Work:                (54)\r\n        Gflops:                       266.412\r\n     Time (ns):                                      199,435\r\n\r\n        Handle:                   2\r\n Child Handles:               (3,4)\r\n        Length:         (540,540,8)\r\n     Placeness:             InPlace\r\n    Input Dist:             2350080\r\n   Output Dist:             1175040\r\n  Input Stride:      (1,544,293760)\r\n Output Stride:      (1,272,146880)\r\n        Gflops:                       549.326\r\n     Time (ns):                                      385,461\r\n\r\n        Handle:                   5\r\n        Kernel:    00000000042069E0\r\n     OutEvents:    0000000005380590\r\n        Length:         (8,271,540)\r\n     Placeness:             InPlace\r\n    Input Dist:             1175040\r\n   Output Dist:             1175040\r\n  Input Stride:      (146880,1,272)\r\n Output Stride:      (146880,1,272)\r\n   Global Work:            (292736)\r\n    Local Work:                (64)\r\n        Gflops:                        225.79\r\n     Time (ns):                                       77,774\r\n\r\n        Handle:                   1\r\n Child Handles:               (2,5)\r\n        Length:         (540,540,8)\r\n     Placeness:             InPlace\r\n    Input Dist:             2350080\r\n   Output Dist:             1175040\r\n  Input Stride:      (1,544,293760)\r\n Output Stride:      (1,272,146880)\r\n        Gflops:                       532.636\r\n     Time (ns):                                      463,235\r\n```\r\n\r\n\r\nI tested this on both 2.8.0 and 2.10.1 - same behaviour. If I drop the Z-dimension similar thing happens. Furthermore this happens also for other sizes than 540, provided that the Y-stride = FFT_X + 2. For power of 2 FFT sizes the slowdown is about 2x.\r\n\r\nAlso - it is not a particular stride that triggers this behaviour (so it's probably not due to memory coalescing?). It can't be channel conflicts either, because those happen for power of two strides. If I use the same strides as above for 500 x 500 x 8 transform I get good performance, but if I now change the stride to 502 I again get the 3x slowdown.\r\n\r\nIf i use the -d option to dump the kernels I can see that in the second case there are no transposition kernels generated and overall the profile info shows fewer handles in the second case, so this is probably the reason. Does anybody know why? Is there some relationship between the FFT size and its strides that results in optimal performance? I'd like to understand it, so I can choose the optimal stride for my application.\r\n\r\nThanks,\r\nMichal\r\n"},"comment":{"url":"https://api.github.com/repos/clMathLibraries/clFFT/issues/comments/204045323","html_url":"https://github.com/clMathLibraries/clFFT/issues/139#issuecomment-204045323","issue_url":"https://api.github.com/repos/clMathLibraries/clFFT/issues/139","id":204045323,"user":{"login":"bragadeesh","id":4991048,"avatar_url":"https://avatars.githubusercontent.com/u/4991048?v=3","gravatar_id":"","url":"https://api.github.com/users/bragadeesh","html_url":"https://github.com/bragadeesh","followers_url":"https://api.github.com/users/bragadeesh/followers","following_url":"https://api.github.com/users/bragadeesh/following{/other_user}","gists_url":"https://api.github.com/users/bragadeesh/gists{/gist_id}","starred_url":"https://api.github.com/users/bragadeesh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bragadeesh/subscriptions","organizations_url":"https://api.github.com/users/bragadeesh/orgs","repos_url":"https://api.github.com/users/bragadeesh/repos","events_url":"https://api.github.com/users/bragadeesh/events{/privacy}","received_events_url":"https://api.github.com/users/bragadeesh/received_events","type":"User","site_admin":false},"created_at":"2016-03-31T17:39:52Z","updated_at":"2016-03-31T17:39:52Z","body":"Hi Michal,\r\n\r\nThanks for using our library. We have not done full performance characterization for all the combinations of transform sizes and strides. The presence of transposes helps in many cases and does not in others; obviously in your use case the transpose kernels are being used but is negatively affecting performance. If you can build the library from source, then I can suggest changes in the code to turn off transposes. It is always best to use strides and distances that resemble a 'packed' format.\r\n\r\nIn master branch, in file: src/library/plan.cpp; line 2496 (for 2D forward real transform) and line 2791 (for 2D inverse real transform) there are if statements. If you change the code so that they evaluate to false, you will disable transposes. Similarly for 3D real transforms, you can change lines 3402 and 3701 the same way.\r\n\r\n\r\n"}}