{"action":"created","issue":{"url":"https://api.github.com/repos/golemfactory/concent/issues/865","repository_url":"https://api.github.com/repos/golemfactory/concent","labels_url":"https://api.github.com/repos/golemfactory/concent/issues/865/labels{/name}","comments_url":"https://api.github.com/repos/golemfactory/concent/issues/865/comments","events_url":"https://api.github.com/repos/golemfactory/concent/issues/865/events","html_url":"https://github.com/golemfactory/concent/issues/865","id":366360554,"node_id":"MDU6SXNzdWUzNjYzNjA1NTQ=","number":865,"title":"Implementation changes for Concent to support processing more than one use case at the same time","user":{"login":"rwrzesien","id":10088909,"node_id":"MDQ6VXNlcjEwMDg4OTA5","avatar_url":"https://avatars0.githubusercontent.com/u/10088909?v=4","gravatar_id":"","url":"https://api.github.com/users/rwrzesien","html_url":"https://github.com/rwrzesien","followers_url":"https://api.github.com/users/rwrzesien/followers","following_url":"https://api.github.com/users/rwrzesien/following{/other_user}","gists_url":"https://api.github.com/users/rwrzesien/gists{/gist_id}","starred_url":"https://api.github.com/users/rwrzesien/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rwrzesien/subscriptions","organizations_url":"https://api.github.com/users/rwrzesien/orgs","repos_url":"https://api.github.com/users/rwrzesien/repos","events_url":"https://api.github.com/users/rwrzesien/events{/privacy}","received_events_url":"https://api.github.com/users/rwrzesien/received_events","type":"User","site_admin":false},"labels":[{"id":879729245,"node_id":"MDU6TGFiZWw4Nzk3MjkyNDU=","url":"https://api.github.com/repos/golemfactory/concent/labels/acceptance%20use%20case","name":"acceptance use case","color":"db9e59","default":false},{"id":822556027,"node_id":"MDU6TGFiZWw4MjI1NTYwMjc=","url":"https://api.github.com/repos/golemfactory/concent/labels/feature","name":"feature","color":"f2dc15","default":false},{"id":856693158,"node_id":"MDU6TGFiZWw4NTY2OTMxNTg=","url":"https://api.github.com/repos/golemfactory/concent/labels/not%20in%20pivotal","name":"not in pivotal","color":"13ada0","default":false},{"id":879728898,"node_id":"MDU6TGFiZWw4Nzk3Mjg4OTg=","url":"https://api.github.com/repos/golemfactory/concent/labels/payment%20use%20case","name":"payment use case","color":"c0e5f7","default":false},{"id":879728704,"node_id":"MDU6TGFiZWw4Nzk3Mjg3MDQ=","url":"https://api.github.com/repos/golemfactory/concent/labels/report%20use%20case","name":"report use case","color":"bff98b","default":false},{"id":879728318,"node_id":"MDU6TGFiZWw4Nzk3MjgzMTg=","url":"https://api.github.com/repos/golemfactory/concent/labels/result%20transfer%20use%20case","name":"result transfer use case","color":"f9d0c4","default":false},{"id":898388062,"node_id":"MDU6TGFiZWw4OTgzODgwNjI=","url":"https://api.github.com/repos/golemfactory/concent/labels/sync%20with%20deployment","name":"sync with deployment","color":"FF1493","default":false},{"id":879727795,"node_id":"MDU6TGFiZWw4Nzk3Mjc3OTU=","url":"https://api.github.com/repos/golemfactory/concent/labels/verification%20use%20case","name":"verification use case","color":"f2f296","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-03T14:13:10Z","updated_at":"2018-10-05T08:59:52Z","closed_at":null,"author_association":"COLLABORATOR","body":"Implementation changes required for Concent to process more than one use case at a time:\r\n\r\n#### Constants\r\n\r\n- Add enum `ConcentUseCase` containing list of use cases handled by Concent.\r\n\r\n`ConcentUseCase` enum:\r\n- ForcedReport\r\n- ForcedTaskResul\r\n- ForcedAcceptance\r\n- AdditionalVerification\r\n- ForcedPayment\r\n\r\n#### Storage database models\r\n\r\n- Add `SubtaskState` model:\r\n\r\n| Column name                   | Type                                            | Remarks                                                                                                                 |\r\n|-------------------------------|-------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|\r\n| `subtask_id` | string | Fields `subtask_id` and `use_case` must be unique together. Must not be NULL or blank.\r\n| `use_case` | enum | Field which will contain value from `ConcentUseCase` enum. Must not be NULL or blank.\r\n| `state` | enum | The same as currently on `Subtask` model. Must not be NULL or blank.\r\n| `next_deadline` | datetime | If in an active state, it's the time at which Concent automatically transitions to a passive state if it does not get the information it expects from the client \t\t\t\t\t\t(a timeout). Must be NULL in a passive state.\r\n\r\n- Remove `state` column from `Subtask` model.\r\n- Move validation of states transitions and `next_deadline` from `Subtask` model `clean` method to `SubtaskState` model `clean` method.\r\n- Refactor validation of related messages against current state to check `SubtaskState` objects related to given `Subtask`.\r\n\r\n#### Data migration\r\n\r\nAccording to changes in storage database models, we should preserve states of currently stored and processed `Subtasks`. To achieve this, migration should be performed in following steps:\r\n\r\n1. Add `SubtaskState` model.\r\n2. Create `SubtaskState` objects from current `Subtask` states. This is the tricky part, because it will require gathering data from several objects to determine if particular use case was processed and what was its results. The verification if given use case was processed should be based on `Subtask` data and existance of related `StoredMessage` and `PendingResponse`.\r\n3. Remove `state` column from `Subtask` model.\r\n\r\n#### Concent core\r\n\r\nThis is the entry point for starting all use cases, so only changes here should be required.\r\n\r\n- `send` handlers should create, update, validate existence of `SubtaskState` objects and their values depending on given use case. The refactored functionality should match current one.\r\n- Modify `update_all_timed_out_subtasks_of_client` to iterate over active `SubtaskState` objects related with given client (#TODO: to achieve this in single query it would be better to add relation between `Subtask` and `SubtaskState` models).\r\n- Modify `store_subtask` and `update_and_return_updated_subtask` functions to not accept parameters `state` and `next_deadline`.\r\n- Create `store_subtask_state` and `update_subtask_state` functions.\r\n\r\n#### Concent core admin\r\n\r\nIt should be possible to see all `SubtaskState` objects and their status (`use_case`, `state`, `next_deadline`) for each `Subtask`. The question is if these information should be visible in main list or in details only.\r\n\r\nFiltering for active/passive subtasks should be refactored. #TODO: I am not sure how this should really work after this change.\r\n\r\n#### Concent soft shutdown\r\n\r\nMailing admins check query should be changed to check if all `SubtaskStates` became passive.\r\n\r\n#### Unit+ tests\r\n\r\nFor most of currently existing tests it should be enough to change test utility method `ConcentIntegrationTestCase._test_subtask_state`. This method should most probably accept value from enum `UseCase` as a parameter, so it will be required to go through all tests which use it and set it. But a shortcut to this might be adding an attribute to `ConcentIntegrationTestCase`, like ie. `self.expected_use_case`, and check it in _test_subtask_state, as in most of the cases each subclass of `ConcentIntegrationTestCase` tests single use case.\r\n\r\nThere are also tests which doesn't inherit from `ConcentIntegrationTestCase` and check Subtask state, so they need to be changed manually.\r\n\r\nTests which create `Subtask` with state does it mostly using `store_subtask` function, so those should be changes accordingly in the same way as in Concent core.\r\n\r\nSome new tests have to be added as well. Those should test different combinations of running use cases at the same time. It should be much easier to achieve this after good unit+ tests refactoring, so the use cases steps will generalized and simplified in calling. The ultimate scenario would be to test processig of all use cases at the same time.\r\n\r\n#### End-to-end tests\r\n\r\nCurrent end-to-end tests process single subtask at a time, so there should be added additional tests which will process different combinations of use cases at the same time.\r\n"},"comment":{"url":"https://api.github.com/repos/golemfactory/concent/issues/comments/427294835","html_url":"https://github.com/golemfactory/concent/issues/865#issuecomment-427294835","issue_url":"https://api.github.com/repos/golemfactory/concent/issues/865","id":427294835,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNzI5NDgzNQ==","user":{"login":"dybi","id":36961162,"node_id":"MDQ6VXNlcjM2OTYxMTYy","avatar_url":"https://avatars0.githubusercontent.com/u/36961162?v=4","gravatar_id":"","url":"https://api.github.com/users/dybi","html_url":"https://github.com/dybi","followers_url":"https://api.github.com/users/dybi/followers","following_url":"https://api.github.com/users/dybi/following{/other_user}","gists_url":"https://api.github.com/users/dybi/gists{/gist_id}","starred_url":"https://api.github.com/users/dybi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dybi/subscriptions","organizations_url":"https://api.github.com/users/dybi/orgs","repos_url":"https://api.github.com/users/dybi/repos","events_url":"https://api.github.com/users/dybi/events{/privacy}","received_events_url":"https://api.github.com/users/dybi/received_events","type":"User","site_admin":false},"created_at":"2018-10-05T08:59:52Z","updated_at":"2018-10-05T08:59:52Z","author_association":"COLLABORATOR","body":"Seems OK, no remarks come to my mind."}}
{"actor":{"display_login":"dybi"}}