{"action":"created","issue":{"url":"https://api.github.com/repos/pyFFTW/pyFFTW/issues/111","repository_url":"https://api.github.com/repos/pyFFTW/pyFFTW","labels_url":"https://api.github.com/repos/pyFFTW/pyFFTW/issues/111/labels{/name}","comments_url":"https://api.github.com/repos/pyFFTW/pyFFTW/issues/111/comments","events_url":"https://api.github.com/repos/pyFFTW/pyFFTW/issues/111/events","html_url":"https://github.com/pyFFTW/pyFFTW/issues/111","id":158022525,"number":111,"title":"Function empty_aligned fails on certain dtypes...","user":{"login":"TrekkieByDay","id":12223749,"avatar_url":"https://avatars.githubusercontent.com/u/12223749?v=3","gravatar_id":"","url":"https://api.github.com/users/TrekkieByDay","html_url":"https://github.com/TrekkieByDay","followers_url":"https://api.github.com/users/TrekkieByDay/followers","following_url":"https://api.github.com/users/TrekkieByDay/following{/other_user}","gists_url":"https://api.github.com/users/TrekkieByDay/gists{/gist_id}","starred_url":"https://api.github.com/users/TrekkieByDay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TrekkieByDay/subscriptions","organizations_url":"https://api.github.com/users/TrekkieByDay/orgs","repos_url":"https://api.github.com/users/TrekkieByDay/repos","events_url":"https://api.github.com/users/TrekkieByDay/events{/privacy}","received_events_url":"https://api.github.com/users/TrekkieByDay/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2016-06-01T22:36:13Z","updated_at":"2016-06-02T11:26:34Z","closed_at":null,"body":"When building the FFT, the function `builders._utils._Xfftn` is eventually called. This calls `builders._utils._compute_array_shapes` to determine the appropriate `input_shape` and `output_shape` and then eventually `pyfftw.empty_aligned(output_shape, output_dtype)`. We were getting the following exception when building FFTs on very large `complex64` arrays:\r\n\r\n    ValueError: negative dimensions are not allowed\r\n\r\nThis was eventually traced to the `builders._compute_array_shapes` call which produced a tuple of `int32` dimensions for the respective shape tuples. I assume they were `int32` as a consequence of working on `complex64` arrays. In any event, the negative dimensions were probably due to integer wrapping for numbers greater than the capacity of `int32`. See the following instructive example, which was carried out using the pre-built `pyFFTW-0.10.1-cp34-none-win_amd64.whl`::\r\n\r\n    >>> dtype = np.dtype(np.complex64)\r\n    >>> shape = (np.int32(2**14), np.int32(2**14))\r\n    >>> pyfftw.empty_aligned(shape, dtype)\r\n    \r\n    ValueError: negative dimensions are not allowed\r\n\r\nAnd a little bit of tweaking...\r\n\r\n    >>> shape = (np.int32(2**14 - 1), np.int32(2**14))\r\n    >>> pyfftw.empty_aligned(shape, dtype)\r\n    array([[ 0.+0.j,  0.+0.j,  0.+0.j, ...,  0.+0.j,  0.+0.j,  0.+0.j],\r\n           [ 0.+0.j,  0.+0.j,  0.+0.j, ...,  0.+0.j,  0.+0.j,  0.+0.j],\r\n           [ 0.+0.j,  0.+0.j,  0.+0.j, ...,  0.+0.j,  0.+0.j,  0.+0.j],\r\n           ..., \r\n           [ 0.+0.j,  0.+0.j,  0.+0.j, ...,  0.+0.j,  0.+0.j,  0.+0.j],\r\n           [ 0.+0.j,  0.+0.j,  0.+0.j, ...,  0.+0.j,  0.+0.j,  0.+0.j],\r\n           [ 0.+0.j,  0.+0.j,  0.+0.j, ...,  0.+0.j,  0.+0.j,  0.+0.j]], dtype=complex64)\r\n\r\nIn order to perform the required FFTs, I changed `builders._compute_array_shapes` so that it **always** casts the shape to a tuple of Python `int`s rather than something based on the provided `dtype`. For us anyway, there does not seem to be a good reason for the shape tuple to be anything other than high capacity integers... the memory footprint is obviously dominated by the `dtype` of the actual array rather than that of its metadata.\r\n\r\nI just wanted to bring this to your attention... I didn't know if this was expected behavior or not. In any event, we did have to make a hack in your package, which we always try to avoid! :)"},"comment":{"url":"https://api.github.com/repos/pyFFTW/pyFFTW/issues/comments/223264832","html_url":"https://github.com/pyFFTW/pyFFTW/issues/111#issuecomment-223264832","issue_url":"https://api.github.com/repos/pyFFTW/pyFFTW/issues/111","id":223264832,"user":{"login":"hgomersall","id":701883,"avatar_url":"https://avatars.githubusercontent.com/u/701883?v=3","gravatar_id":"","url":"https://api.github.com/users/hgomersall","html_url":"https://github.com/hgomersall","followers_url":"https://api.github.com/users/hgomersall/followers","following_url":"https://api.github.com/users/hgomersall/following{/other_user}","gists_url":"https://api.github.com/users/hgomersall/gists{/gist_id}","starred_url":"https://api.github.com/users/hgomersall/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hgomersall/subscriptions","organizations_url":"https://api.github.com/users/hgomersall/orgs","repos_url":"https://api.github.com/users/hgomersall/repos","events_url":"https://api.github.com/users/hgomersall/events{/privacy}","received_events_url":"https://api.github.com/users/hgomersall/received_events","type":"User","site_admin":false},"created_at":"2016-06-02T11:26:34Z","updated_at":"2016-06-02T11:26:34Z","body":"Actually I think the failing might be on line 201.\r\n\r\nSo, can you run the following:\r\n\r\n```python\r\nx = np.zeros((2**14, 2**14), dtype='complex64')\r\nitemsize = x.itemsize\r\n\r\nshape = x.shape\r\nint64_array_length = np.int64(1)\r\norig_array_length = 1\r\nn = 32\r\n\r\nfor each_dimension in shape:\r\n    int64_array_length *= each_dimension\r\n    orig_array_length *= each_dimension\r\n\r\nprint('original: ' + str(orig_array_length))\r\nprint('original new array bytes: ' + str(orig_array_length * itemsize + n))\r\nprint('int64 array: ' + str(int64_array_length))\r\nprint('int64 new array bytes: ' + str(int64_array_length * itemsize + n))\r\nprint('int64 new array bytes modified: ' + str(int64_array_length * np.int64(itemsize) + np.int64(n)))\r\nprint('int64 new array bytes alt: ' + str(np.prod(np.int64(shape)) * itemsize + n))\r\n```"}}
{"actor":{"display_login":"hgomersall"}}