{"action":"created","issue":{"url":"https://api.github.com/repos/golemfactory/concent/issues/139","repository_url":"https://api.github.com/repos/golemfactory/concent","labels_url":"https://api.github.com/repos/golemfactory/concent/issues/139/labels{/name}","comments_url":"https://api.github.com/repos/golemfactory/concent/issues/139/comments","events_url":"https://api.github.com/repos/golemfactory/concent/issues/139/events","html_url":"https://github.com/golemfactory/concent/issues/139","id":299685584,"node_id":"MDU6SXNzdWUyOTk2ODU1ODQ=","number":139,"title":"[Blueprint] Concent state machine","user":{"login":"cameel","id":137030,"node_id":"MDQ6VXNlcjEzNzAzMA==","avatar_url":"https://avatars3.githubusercontent.com/u/137030?v=4","gravatar_id":"","url":"https://api.github.com/users/cameel","html_url":"https://github.com/cameel","followers_url":"https://api.github.com/users/cameel/followers","following_url":"https://api.github.com/users/cameel/following{/other_user}","gists_url":"https://api.github.com/users/cameel/gists{/gist_id}","starred_url":"https://api.github.com/users/cameel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cameel/subscriptions","organizations_url":"https://api.github.com/users/cameel/orgs","repos_url":"https://api.github.com/users/cameel/repos","events_url":"https://api.github.com/users/cameel/events{/privacy}","received_events_url":"https://api.github.com/users/cameel/received_events","type":"User","site_admin":false},"labels":[{"id":822561969,"node_id":"MDU6TGFiZWw4MjI1NjE5Njk=","url":"https://api.github.com/repos/golemfactory/concent/labels/blueprint","name":"blueprint","color":"0052cc","default":false}],"state":"open","locked":false,"assignee":{"login":"cameel","id":137030,"node_id":"MDQ6VXNlcjEzNzAzMA==","avatar_url":"https://avatars3.githubusercontent.com/u/137030?v=4","gravatar_id":"","url":"https://api.github.com/users/cameel","html_url":"https://github.com/cameel","followers_url":"https://api.github.com/users/cameel/followers","following_url":"https://api.github.com/users/cameel/following{/other_user}","gists_url":"https://api.github.com/users/cameel/gists{/gist_id}","starred_url":"https://api.github.com/users/cameel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cameel/subscriptions","organizations_url":"https://api.github.com/users/cameel/orgs","repos_url":"https://api.github.com/users/cameel/repos","events_url":"https://api.github.com/users/cameel/events{/privacy}","received_events_url":"https://api.github.com/users/cameel/received_events","type":"User","site_admin":false},"assignees":[{"login":"cameel","id":137030,"node_id":"MDQ6VXNlcjEzNzAzMA==","avatar_url":"https://avatars3.githubusercontent.com/u/137030?v=4","gravatar_id":"","url":"https://api.github.com/users/cameel","html_url":"https://github.com/cameel","followers_url":"https://api.github.com/users/cameel/followers","following_url":"https://api.github.com/users/cameel/following{/other_user}","gists_url":"https://api.github.com/users/cameel/gists{/gist_id}","starred_url":"https://api.github.com/users/cameel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cameel/subscriptions","organizations_url":"https://api.github.com/users/cameel/orgs","repos_url":"https://api.github.com/users/cameel/repos","events_url":"https://api.github.com/users/cameel/events{/privacy}","received_events_url":"https://api.github.com/users/cameel/received_events","type":"User","site_admin":false}],"milestone":null,"comments":16,"created_at":"2018-02-23T11:44:54Z","updated_at":"2018-07-18T14:52:15Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"### Concent states\r\n\r\nConcent server can be thought of as a state machine with regard to any specific subtask. Upon receiving a message in the `send/` endpoint the processing depends only the current state and the content of the message submitted by the client.\r\n\r\nHere's a diagram that shows all possible Concent states and transitions between them.\r\n\r\n![concent-states-detailed](https://user-images.githubusercontent.com/137030/41709223-f8e16460-7542-11e8-9c2f-60e60385e090.png)\r\n\r\nNote that the states we're talking about here are **Concent states**, not **subtask/task states**. This means that the state reflects Concent's knowledge about the subtask and not necessarily what is happening to it between the provider and requestor. For example if the provider triggers the 'force report computed task' use case, requestor acknowledges it and none of them interact with Concent after that, the subtask stays in the `REPORTED` state even though it may have already been accepted or rejected.\r\n\r\nTo reflect this we can classify the states into two categories:\r\n- **active states** (blue) - states in which the subtask can remain only for a limited time. Concent always eventually exits such a state. Either in response to a message from a client or after a timeout.\r\n- **passive states** (yellow) - states that indicate that Concent is not processing any use case for the subtask. A subtask can remain in such a state indefinitely.\r\n\r\nAnother thing of note is that there are no states for 'forced payment' use case. This use case operates on multiple subtasks and the response is returned immediately so there's no state change here.\r\n\r\n### Subtask table\r\n\r\nTo keep track of subtask states Concent uses database tables with the following structure:\r\n\r\n#### `Subtasks`\r\n\r\n| Column name                   | Type                                            | Remarks                                                                                                                 |\r\n|-------------------------------|-------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|\r\n| `task_id`                     | string                                          | All (`requestor_public_key`, `task_id`) pairs must be globally unique. Must not be `NULL` or blank. A task can have multiple subtasks. Concent always operates on individual subtasks separately. The task ID is kept only for reference.\r\n| `subtask_id`                  | string                                          | All (`requestor_public_key`, `subtask_id`) pairs must be globally unique. Must not be `NULL` or blank.\r\n| `requestor`                   | foreign key to `Client`                         | Must not be `NULL`\r\n| `provider`                    | foreign key to `Client`                         | Must not be `NULL`\r\n| `state`                       | enum (see state diagram)                        | The current Concent state. There is no enumeration value for `UNKNOWN` because it represents a state in which Concent does not know about subtask's existence.\r\n| `next_deadline`               | timestamp                                       | If in an active state, it's the time at which Concent automatically transitions to a passive state if it does not get the information it expects from the client (a timeout). Must be `NULL` in a passive state.\r\n| `computation_deadline`        | timestamp                                       | Same as `task_to_compute.deadline`. Must not be `NULL`.\r\n| `result_package_size`         | int                                             | Same as `report_computed_task.package_size`. `NULL` if and only if `report_computed_task` is `NULL`.\r\n| `task_to_compute`             | foreign key to `StoredTaskToCompute`            | Must not be `NULL`. A subtask always has a `TaskToCompute` associated with it.\r\n| `report_computed_task`        | foreign key to `StoredReportComputedTask`       | Computation report created by the provider. Must not be `NULL`.\r\n| `ack_report_computed_task`    | foreign key to `StoredAckReportComputedTask`    | Subtask acknowledgement to be passed to the provider, either submitted by the requestor or generated by Concent. May be `NULL` if not submitted or generated yet.\r\n| `reject_report_computed_task` | foreign key to `StoredRejectReportComputedTask` | Subtask report rejection to be passed to the provider, either submitted by the requestor or generated by Concent. May be `NULL` if not submitted or generated yet.\r\n| `subtask_results_accepted`    | foreign key to `StoredSubtaskResultsAccepted`   | Subtask acceptance to be passed to the provider, either submitted by the requestor or generated by Concent. May be `NULL` if not submitted or generated yet.\r\n| `subtask_results_rejected`    | foreign key to `StoredSubtaskResultsRejected`   | Subtask rejection to be passed to the provider, either submitted by the requestor or generated by Concent. May be `NULL` if not submitted or generated yet.\r\n\r\n\r\n#### `Clients`\r\n\r\n| Column name                   | Type                                            | Remarks                  |\r\n|-------------------------------|-------------------------------------------------|--------------------------|\r\n| `public_key`                  | string (base64-encoded public key)              | Must not be `NULL` or blank.\r\n\r\n`StoredTaskToCompute`, `StoredReportComputedTask`, `StoredAckReportComputedTask`, `StoredRejectReportComputedTask`, `StoredSubtaskResultsAccepted`, `StoredSubtaskResultsRejected` are all tables storing the content of the corresponding Golem messages. Each table has columns for timestamp, signature and all payload fields. Nested data is represented in a relational way (e.g. nested messages should be foreign keys to the corresponding tables). We want to be able to easily and efficiently use message data in queries. We don't want to have to fetch and decode each message individually. We also want to strictly enforce the schema so that we can easily detect changes and provide data migrations for old data.\r\n\r\n### State transitions\r\n\r\nState transitions can happen in two ways:\r\n- In response to a message submitted to the `send/` endpoint.\r\n- Automatically, after a timeout.\r\n\r\nAutomatic transitions require Concent to actively check subtask states in the background and take actions when the time is up. Note however that even if Concent notices a timeout, the client won't until it receives a response. Due to the nature of the current implementation (Concent is passive and clients query the `receive/` endpoint to get responses) it's possible to achieve the same effect (from client's standpoint) by postponing timeout checks until the client actually asks. That's the approach we're taking for now.\r\n\r\nSo this means that automatic transitions can happen whenever the client contacts one of the three message-related endpoints: `send/`, `receive/` or `receive-out-of-band/`. Before doing any processing in these endpoints Concent runs a function that checks timeouts for all subtasks in which the client is involved, updates the subtask table and adds responses to the receive queues.\r\n\r\nWhen a timeout is detected Concent may have to perform additional steps to determine the outcome. For example it may have to check if all necessary files have been uploaded. The response added to the queue can depend on the result but once it's in the queue it won't be changed or be automatically discarded.\r\n\r\n### Receive queues\r\n\r\nEach client has two queues associated with it:\r\n- receive queue\r\n- receive-out-of-band queue\r\n\r\nEach of them stores information about messages to be returned from the corresponding endpoint. Messages are generated from that data on the fly so that the timestamp is always current. This applies only to the outer messages generated by Concent. If there are nested messages submitted from the outside, their timestamps and signatures are guaranteed to be preserved.\r\n\r\nEvery queue is associated with a single client and can contain messages associated with any of its subtasks.\r\n\r\nMessages are added to the queue as a result of state transitions (see above). Once queued they do not undergo any further processing. They just sit in the queue until the authorized client fetches them.\r\n\r\nMessages are returned in the order of their timestamps (oldest first). Note that, due to the nature of the HTTP protocol, if a client fetches multiple messages in parallel the order in which they reach it is not in any way guaranteed even if the server returns them in a specific order.\r\n\r\nMessages are never discarded from the queue, even after state transitions. It's the responsibility of the client to decide whether a message is still relevant or not.\r\n\r\nA message is returned only once and then marked as delivered. Delivered messages may remain stored on the server for use in further use cases or for audit purposes but there's no way for the client to fetch them again.\r\n\r\n#### Queue internals\r\n\r\nThe queues are implemented using a single database table with locks to ensure that the process of retrieving a message is atomic.\r\n\r\nThe structure of the table is as follows:\r\n\r\n##### `ReceiveQueue`\r\n\r\n| Column name                   | Type                                            | Remarks                  |\r\n|-------------------------------|-------------------------------------------------|--------------------------|\r\n| `response_type`               | `ConcentResponseType` enum                      |\r\n| `client`                      | foreign key to `Client`                         |\r\n| `queue`                       | enum: `receive` or `receive-out-of-band`        |\r\n| `delivered`                   | bool                                            | `TRUE` if the client has already fetched the message.\r\n| `subtask`                     | foreign key to `Subtask`                        |\r\n| `created_at`                  | timestamp                                       | The time at which the message has been added to the queue. This is **not** the timestamp of the final message. The timestamp should be the same as the time at which the Golem message object is created, just before delivery.\r\n\r\nAll the columns must not be `NULL`.\r\n\r\n##### `ConcentResponseType` enumeration\r\n\r\n- `ForceReportComputedTask`\r\n- `ForceReportComputedTaskResponse`\r\n- `VerdictReportComputedTask`\r\n- `ForceGetTaskResultRejected`\r\n- `ForceGetTaskResultFailed`\r\n- `ForceGetTaskResultUpload`\r\n- `ForgetGetTaskResultDownload`\r\n- `ForceSubtaskResults`\r\n- `SubtaskResultsSettled`\r\n- `ForceSubtaskResultsResponse`\r\n- `SubtaskResultsRejected`\r\n- `ForcePaymentCommitted`\r\n\r\nNote that this list contains only those messages that can be added to the receive queue, not all possible messages. Messages that are always returned synchronously from the `send/` endpoint are not included here.\r\n\r\nThe type and the data stored in the `Subtask` table is everything that's needed to create a response. For example if the response is a wrapper for the actual response passed from the requestor, the link to requestor's response is stored in the `Subtask` table. If the content of the response depends on the result of some external checks done after a timeout, this result should be stored in `Subtasks`. The `Subtasks` table (along with the related tables) is the central place to store everything we need to create a complete response. The information should be stored in a relational form - in database columns - rather than inside serialized Golem message objects.\r\n\r\n### Message storage\r\n\r\nEventually every Golem message handled by Concent should have a corresponding database table. In short term only the most often used messages will get their own tables and the rarely used ones will still be stored as binary blobs in the `StoredMessage` table.\r\n\r\nMessage tables are meant to store all messages no matter what their purpose is. Other mechanisms like subtask table, receive queues or audit storage can build on that by referencing them with foreign keys. None of them should make an assumption that stored messages are for their exclusive use.\r\n"},"comment":{"url":"https://api.github.com/repos/golemfactory/concent/issues/comments/405959408","html_url":"https://github.com/golemfactory/concent/issues/139#issuecomment-405959408","issue_url":"https://api.github.com/repos/golemfactory/concent/issues/139","id":405959408,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNTk1OTQwOA==","user":{"login":"MichalW-CP","id":36954871,"node_id":"MDQ6VXNlcjM2OTU0ODcx","avatar_url":"https://avatars3.githubusercontent.com/u/36954871?v=4","gravatar_id":"","url":"https://api.github.com/users/MichalW-CP","html_url":"https://github.com/MichalW-CP","followers_url":"https://api.github.com/users/MichalW-CP/followers","following_url":"https://api.github.com/users/MichalW-CP/following{/other_user}","gists_url":"https://api.github.com/users/MichalW-CP/gists{/gist_id}","starred_url":"https://api.github.com/users/MichalW-CP/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MichalW-CP/subscriptions","organizations_url":"https://api.github.com/users/MichalW-CP/orgs","repos_url":"https://api.github.com/users/MichalW-CP/repos","events_url":"https://api.github.com/users/MichalW-CP/events{/privacy}","received_events_url":"https://api.github.com/users/MichalW-CP/received_events","type":"User","site_admin":false},"created_at":"2018-07-18T14:52:15Z","updated_at":"2018-07-18T14:52:15Z","author_association":"COLLABORATOR","body":"Please add computation_deadline and result_package_size to Subtaski"}}
{"actor":{"display_login":"MichalW-CP"}}