{"action":"opened","issue":{"url":"https://api.github.com/repos/dotnet/corefx/issues/30971","repository_url":"https://api.github.com/repos/dotnet/corefx","labels_url":"https://api.github.com/repos/dotnet/corefx/issues/30971/labels{/name}","comments_url":"https://api.github.com/repos/dotnet/corefx/issues/30971/comments","events_url":"https://api.github.com/repos/dotnet/corefx/issues/30971/events","html_url":"https://github.com/dotnet/corefx/issues/30971","id":340027235,"node_id":"MDU6SXNzdWUzNDAwMjcyMzU=","number":30971,"title":"Resolve un-contiguous memory into contiguous memory","user":{"login":"msedi","id":8020659,"node_id":"MDQ6VXNlcjgwMjA2NTk=","avatar_url":"https://avatars0.githubusercontent.com/u/8020659?v=4","gravatar_id":"","url":"https://api.github.com/users/msedi","html_url":"https://github.com/msedi","followers_url":"https://api.github.com/users/msedi/followers","following_url":"https://api.github.com/users/msedi/following{/other_user}","gists_url":"https://api.github.com/users/msedi/gists{/gist_id}","starred_url":"https://api.github.com/users/msedi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msedi/subscriptions","organizations_url":"https://api.github.com/users/msedi/orgs","repos_url":"https://api.github.com/users/msedi/repos","events_url":"https://api.github.com/users/msedi/events{/privacy}","received_events_url":"https://api.github.com/users/msedi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-10T22:20:51Z","updated_at":"2018-07-10T22:20:51Z","closed_at":null,"author_association":"NONE","body":"As suggested I copied my issue #2297 from corefxlab to here:\r\n\r\nI'm often having the problem that I have to deal with large image volumes. Currently to run quickly over a volume and do image operations on the data, I need to copy the whole volume into one large block of memory. Assume that I have a volume with a matrix size of 512x512 and around 1000 images stored in float, I have doubled the amount of memory in my system only because I need the data contiguously in memory.\r\n\r\nI don't know how to resolve this, but it would be convenient to a structure that merges all data into one let's say Span.\r\n\r\nI'm \"dreaming\" of something that can be done like this:\r\n\r\n```\r\nfloat[] A = new float[1000];\r\nfloat[] B = new float[1000];\r\n\r\nSpan<float> C = Span<T>.Merge(A, B);\r\n```\r\n\r\nwhere Span.Merge is defined as\r\n\r\n`static Span<T> Merge(params Span<T>[] spans)`\r\n\r\nIn the end, only the internal indexer needs to take care of going into the right span when doing something like this (example from above)\r\n\r\n```\r\nfloat[] A = new float[1000];\r\nfloat[] B = new float[1000];\r\n\r\nSpan<float> C = Span<T>.Merge(A, B);\r\nfor (int i=0; i<C.Length; i++) C[i] += 10;\r\n```\r\nOf course this can be managed by myself, but since Span is a struct, I'm not able to do it myself.\r\n\r\nThat would be awesome. To make it more clear I made some example that shows where the need is:\r\nJust to clarify a few terms in advance.\r\nWhen I'm talking about an image volume, I mean a 3D volume that consists of a stack of individual slices. \r\n\r\nTo wrap the individual slices I have created a container called `VoxelVolume`. The `VoxelVolume `has an indexer `VoxelVolume[int sliceIndex]` that returns a `VolumeSlice `class. Each VolumeSlice has a method where I can get access to the data:\r\n\r\n-  `float[] GetData(AreaDescriptor ad = null)`\r\nThat returns the internal data of the slice either completely, or only from a certain area in the full slice.\r\n\r\n- `void SetData(float[] data, AreaDescriptor = null)`\r\nThat sets the full data of to the slice or only a portion.\r\n\r\nAlso to clarify a few things, why it's not possible to access the data of the slice directly is first, because there are additional parameter on the slice, like orientation, etc. Another thing is that the slice can unload itself if the memory consumption of the whole system is getting low.\r\n\r\nThere are now occasions where I need to filter the data. One pretty easy thing would be to create a mean value for each pixel of the image in the surrounding area, first, of course in plane (x/y-direction), but also out of the plane (z-direction). To be fast I'm using pointers, but for the sake of brevity I leave this one out in the example below. Assuming a few simplifications that the volume is divisible by 10 and I don't care about the edges, although this would produce an error.\r\n\r\n```\r\n        VoxelVolume Filter(VoxelVolume volume, int filterSize)\r\n        {\r\n            VoxelVolume result = volume.Clone() as VoxelVolume;\r\n\r\n\r\n            float[] aBuffer = new float[(filterSize * 2 + 1) * volume.NxNy];\r\n            float[] aResult = new float[volume.NxNy];\r\n\r\n            // Run over the slices.\r\n            for (int SliceNo = 0; SliceNo < volume.Length; SliceNo++)\r\n            {\r\n                // Copy the data in a block\r\n                int aTargetIndex = 0;\r\n                for (int aSourceNo = SliceNo - filterSize; aSourceNo < SliceNo + filterSize; aSourceNo++)\r\n                {\r\n                    Array.Copy(volume[aSourceNo].GetData(), 0, aBuffer, aTargetIndex, volume.NxNy);\r\n                    aTargetIndex += volume.NxNy;\r\n                }\r\n\r\n                double aSum = 0;\r\n\r\n                for (int X = 0; X < volume.Nx; X++)\r\n                for (int Y = 0; Y < volume.Ny; Y++)\r\n                {\r\n                    for (int z = 0; z < (filterSize * 2) + 1; z++)\r\n                    for (int x = X - (filterSize * 2) + 1; x < X + (filterSize * 2) + 1; x++)\r\n                    for (int y = Y - (filterSize * 2) + 1; y < Y + (filterSize * 2) + 1; y++)\r\n                    {\r\n                        aSum += aBuffer[(z * volume.NxNy) + (y * volume.Nx) + x];\r\n                    }\r\n\r\n                    aResult[(Y * volume.Nx) + X] = aSum / (double) ((filterSize * 2 + 1));\r\n                }\r\n\r\n                result[aSliceNo].SetData(aResult);\r\n            }\r\n        }\r\n```\r\n\r\nIf course this works this way, but the final solution is that for performance reasons I also put this into a Parallel.For which consumes a lot of memory. I was hoping there would be a solution that abstracts the memory in a way I don't have to copy but also has the performance of the example above. BufferSegment would be interesting, but I still have to deal with the inidividual segments. Something like the code below would be good:\r\n\r\n```\r\n        VoxelVolume Filter(VoxelVolume volume, int filterSize)\r\n        {\r\n            VoxelVolume result = volume.Clone() as VoxelVolume;\r\n\r\n\r\n            float[] aResult = new float[volume.NxNy];\r\n\r\n            // Run over the slices.\r\n            for (int SliceNo = 0; SliceNo < volume.Length; SliceNo++)\r\n            {\r\n                BufferSegment<float> aBufferSegment = new BufferSegment<>();\r\n\r\n                // Copy the data in a block\r\n                int aTargetIndex = 0;\r\n                for (int aSourceNo = SliceNo - filterSize; aSourceNo < SliceNo + filterSize; aSourceNo++)\r\n                {\r\n                    aBufferSegment.Add(volume[aSourceNo].GetData());\r\n                    aTargetIndex += volume.NxNy;\r\n                }\r\n\r\n                Span<float> aMemoryBlock = aBufferSegment.Span;\r\n\r\n                double aSum = 0;\r\n\r\n                for (int X = 0; X < volume.Nx; X++)\r\n                for (int Y = 0; Y < volume.Ny; Y++)\r\n                {\r\n                    for (int z = 0; z < (filterSize * 2) + 1; z++)\r\n                    for (int x = X - (filterSize * 2) + 1; x < X + (filterSize * 2) + 1; x++)\r\n                    for (int y = Y - (filterSize * 2) + 1; y < Y + (filterSize * 2) + 1; y++)\r\n                    {\r\n                        aSum += aMemoryBlock[(z * volume.NxNy) + (y * volume.Nx) + x];\r\n                    }\r\n\r\n                    aResult[(Y * volume.Nx) + X] = aSum / (double) ((filterSize * 2 + 1));\r\n                }\r\n\r\n                result[aSliceNo].SetData(aResult);\r\n            }\r\n```\r\n\r\nDoes this makes sense?\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n"}}
{"actor":{"display_login":"msedi"}}