{"action":"created","issue":{"url":"https://api.github.com/repos/KhronosGroup/Vulkan-Docs/issues/755","repository_url":"https://api.github.com/repos/KhronosGroup/Vulkan-Docs","labels_url":"https://api.github.com/repos/KhronosGroup/Vulkan-Docs/issues/755/labels{/name}","comments_url":"https://api.github.com/repos/KhronosGroup/Vulkan-Docs/issues/755/comments","events_url":"https://api.github.com/repos/KhronosGroup/Vulkan-Docs/issues/755/events","html_url":"https://github.com/KhronosGroup/Vulkan-Docs/issues/755","id":345834211,"node_id":"MDU6SXNzdWUzNDU4MzQyMTE=","number":755,"title":"Device - Host - Device synchronization with VkEvent","user":{"login":"neXyon","id":2699957,"node_id":"MDQ6VXNlcjI2OTk5NTc=","avatar_url":"https://avatars1.githubusercontent.com/u/2699957?v=4","gravatar_id":"","url":"https://api.github.com/users/neXyon","html_url":"https://github.com/neXyon","followers_url":"https://api.github.com/users/neXyon/followers","following_url":"https://api.github.com/users/neXyon/following{/other_user}","gists_url":"https://api.github.com/users/neXyon/gists{/gist_id}","starred_url":"https://api.github.com/users/neXyon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/neXyon/subscriptions","organizations_url":"https://api.github.com/users/neXyon/orgs","repos_url":"https://api.github.com/users/neXyon/repos","events_url":"https://api.github.com/users/neXyon/events{/privacy}","received_events_url":"https://api.github.com/users/neXyon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2018-07-30T16:28:50Z","updated_at":"2018-08-09T13:55:26Z","closed_at":null,"author_association":"NONE","body":"Shouldn't this be possible according to the specification? Here is a code sample:\r\n\r\nhttps://gist.github.com/neXyon/859b2e52bac9a5a56b804d8a9d5fa4a5\r\n\r\nI'm trying to synchronize a host stage into my pipeline, where I basically edit some data on the host during the execution of a command buffer on the device. From reading the specification I think I'm doing the correct synchronization, execution/memory dependencies and availability/visibility operations, but it neither works on NV nor AMD hardware. It works on Adreno though. I produced a minimal example (see link above) that shows how I'm trying to implement it:\r\n\r\n1. A buffer is filled with increasing integers and uploaded to the device (device local buffer).\r\n2. This buffer is copied to a host visible one.\r\n3. The first event is set.\r\n4. The second event is waited for.\r\n5. Meanwhile the host waits for the first event.\r\n6. After it has been set, it increments the numbers in the host visible buffer.\r\n7. Then it sets the second event.\r\n8.  The device then continues to copy the host visible buffer back to the device local buffer.\r\n9. Finally, this buffer is copied again to a host visible one for checking.\r\n\r\n### What happens?\r\n\r\nOn NV the first part works, the correct data arrives at the host side, but the altered data never arrives at the device side. On AMD not even the first part works and I already don't get the data on the host.\r\n\r\n### The buffers:\r\n\r\n- device_buffer is a buffer with VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT memory and VK_BUFFER_USAGE_TRANSFER_DST_BIT | VK_BUFFER_USAGE_TRANSFER_SRC_BIT usage\r\n- host_buffer is a buffer with VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT memory and VK_BUFFER_USAGE_TRANSFER_DST_BIT | VK_BUFFER_USAGE_TRANSFER_SRC_BIT usage\r\n- result_buffer is a buffer with VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT memory and VK_BUFFER_USAGE_TRANSFER_DST_BIT usage\r\n"},"comment":{"url":"https://api.github.com/repos/KhronosGroup/Vulkan-Docs/issues/comments/411766043","html_url":"https://github.com/KhronosGroup/Vulkan-Docs/issues/755#issuecomment-411766043","issue_url":"https://api.github.com/repos/KhronosGroup/Vulkan-Docs/issues/755","id":411766043,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMTc2NjA0Mw==","user":{"login":"drakos-amd","id":9551469,"node_id":"MDQ6VXNlcjk1NTE0Njk=","avatar_url":"https://avatars0.githubusercontent.com/u/9551469?v=4","gravatar_id":"","url":"https://api.github.com/users/drakos-amd","html_url":"https://github.com/drakos-amd","followers_url":"https://api.github.com/users/drakos-amd/followers","following_url":"https://api.github.com/users/drakos-amd/following{/other_user}","gists_url":"https://api.github.com/users/drakos-amd/gists{/gist_id}","starred_url":"https://api.github.com/users/drakos-amd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drakos-amd/subscriptions","organizations_url":"https://api.github.com/users/drakos-amd/orgs","repos_url":"https://api.github.com/users/drakos-amd/repos","events_url":"https://api.github.com/users/drakos-amd/events{/privacy}","received_events_url":"https://api.github.com/users/drakos-amd/received_events","type":"User","site_admin":false},"created_at":"2018-08-09T13:55:26Z","updated_at":"2018-08-09T13:55:26Z","author_association":"NONE","body":"Fine-grained mid-command buffer synchronization between the host and the device should, in theory, be possible in the way you propose if my understanding it correct. Let's go through the individual steps:\r\n\r\n> 1. A buffer is filled with increasing integers and uploaded to the device (device local buffer).\r\n> 2. This buffer is copied to a host visible one.\r\n> 3. The first event is set.\r\n\r\nAs already discussed above, if there's a device-to-host memory or execution dependency then the `vkCmdSetEvent` call should be preceeded by a `vkCmdPipelineBarrier` with its `dstStageMask` parameter set to the same value as the `stageMask` parameter specified in `vkCmdSetEvent` in order to ensure that you have the proper happens-before relation.\r\n\r\nIf there's a memory dependency then obviously `dstAccessMask` should include `VK_ACCESS_HOST_READ_BIT` and/or `VK_ACCESS_HOST_WRITE_BIT` (depending on what you do with the shared data, in your example it looks like only the latter is needed).\r\n\r\nIn the simplest case when there's no device-to-host memory or execution dependency then this stage mask could be simply `VK_PIPELINE_STAGE_TOP_OF_PIPE`, and no `vkCmdPipelineBarrier` call is needed, but I assume you have some dependency otherwise you wouldn't need the first event.\r\n\r\nAlso, if there's only an execution dependency but no memory dependency then you can simply specify the appropriate `stageMask` to signal-after directly to `vkCmdSetEvent` (no need for `vkCmdPipelineBarrier`).\r\n\r\n(Note that there's no reason/need for accepting `VK_PIPELINE_STAGE_HOST_BIT` in `vkCmdSetEvent` contrarily to what has been suggested in a previous comment as `vkCmdSetEvent` specifies the pipeline stages after which the event should be set by the device, not the stage where the event will be subsequently waited for.)\r\n\r\n> 4. The second event is waited for.\r\n\r\nIn order to have proper synchronization, the following parameters have to be specified to `vkCmdWaitEvents`:\r\n- `srcStageMask` must be `VK_PIPELINE_STAGE_HOST_BIT`, as we're waiting for an event signaled by the host\r\n- `srcAccessMask` of the memory barrier has to include `VK_ACCESS_HOST_WRITE_BIT`, as you're waiting for the host writes to become visible\r\n\r\nThe values of `dstStageMask` and `dstAccessMask` evidently will have to be set according to what you plan to do with the shared resource on the device after the wait.\r\n\r\n> 5. Meanwhile the host waits for the first event.\r\n\r\nSure, spinning until `vkGetEventStatus` returns `VK_EVENT_SET` is fine.\r\n\r\n> 6. After it has been set, it increments the numbers in the host visible buffer.\r\n\r\nOne thing to pay attention when reading/writing shared data in host-mapped allocations is that if the memory type of the allocation does not have the `VK_MEMORY_PROPERTY_HOST_COHERENT_BIT` property then you have to use `vkInvalidateMappedMemoryRanges` and/or `vkFlushMappedMemoryRanges` if you read and/or write the memory, respectively, to ensure that coherency between the host accesses and device accesses synchronized with `VK_ACCESS_HOST_READ_BIT`/`VK_ACCESS_HOST_WRITE_BIT`.\r\n\r\nFrom your description it looks like you do use a memory type that has the `VK_MEMORY_PROPERTY_HOST_COHERENT_BIT` but I'd suggest you to give a try issuing an invalidate/flush for debugging purposes to see if it changes anything.\r\n\r\nI also wonder whether using a memory type with or without `VK_MEMORY_PROPERTY_HOST_CACHED_BIT` changes anything.\r\n\r\n> 7. Then it sets the second event.\r\n> 8. The device then continues to copy the host visible buffer back to the device local buffer.\r\n\r\nConsidering the next operation you do with the buffer is a copy, it means the parameters of the `vkCmdWaitEvents` call in step 4 has to have:\r\n- `dstStageMask` set to `VK_PIPELINE_STAGE_TRANSFER_BIT`\r\n- `dstAccessMask` set to `VK_ACCESS_TRANSFER_READ_BIT`\r\n\r\nThe parameters you specified for `vkCmdWaitEvents` seem to match the above.\r\n\r\nI did try out your example code on github on an AMD GPU and it seems to work just fine (no mismatches in the expected results).\r\n\r\nIn your example code the stage mask parameters of `vkCmdSetEvent` and its preceeding `vkCmdPipelineBarrier` are a bit superfluous, and in fact incorrect, as the `dstStageMask` parameter of `vkCmdPipelineBarrier` must be a strict superset of the `stageMask` parameter of `vkCmdSetEvent` as explained earlier, though in practice it happens to work as is (as implementations may map multiple API pipeline stages to the same underlying HW-specific value). The correct and minimal stage masks would be to:\r\n- set the `srcStageMask` parameter of `vkCmdPipelineBarrier` to `VK_PIPELINE_STAGE_TRANSFER_BIT` (as you have a transfer preceeding the barrier), and\r\n- set the `dstStageMask` parameter of `vkCmdPipelineBarrier` and the `stageMask` parameter of `vkCmdSetEvent` to the same value (e.g. `VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT`)\r\n\r\nIndifferently though either with the above modifications or with the original (strictly speaking incorrect) parameters the test seem to work just fine on the configuration I've tested it on.\r\n\r\nCan you provide more information on what AMD GPU, driver version, OS, and architecture (32-bit or 64-bit build) did you see the incorrect behavior on?"}}
{"actor":{"display_login":"drakos-amd"}}