{"action":"created","issue":{"url":"https://api.github.com/repos/Cloud-CV/EvalAI/issues/841","repository_url":"https://api.github.com/repos/Cloud-CV/EvalAI","labels_url":"https://api.github.com/repos/Cloud-CV/EvalAI/issues/841/labels{/name}","comments_url":"https://api.github.com/repos/Cloud-CV/EvalAI/issues/841/comments","events_url":"https://api.github.com/repos/Cloud-CV/EvalAI/issues/841/events","html_url":"https://github.com/Cloud-CV/EvalAI/issues/841","id":214607135,"node_id":"MDU6SXNzdWUyMTQ2MDcxMzU=","number":841,"title":"Publish message whenever file field of Challenge and Challenge Phase changes","user":{"login":"taranjeet","id":4302268,"node_id":"MDQ6VXNlcjQzMDIyNjg=","avatar_url":"https://avatars2.githubusercontent.com/u/4302268?v=4","gravatar_id":"","url":"https://api.github.com/users/taranjeet","html_url":"https://github.com/taranjeet","followers_url":"https://api.github.com/users/taranjeet/followers","following_url":"https://api.github.com/users/taranjeet/following{/other_user}","gists_url":"https://api.github.com/users/taranjeet/gists{/gist_id}","starred_url":"https://api.github.com/users/taranjeet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/taranjeet/subscriptions","organizations_url":"https://api.github.com/users/taranjeet/orgs","repos_url":"https://api.github.com/users/taranjeet/repos","events_url":"https://api.github.com/users/taranjeet/events{/privacy}","received_events_url":"https://api.github.com/users/taranjeet/received_events","type":"User","site_admin":false},"labels":[{"id":478261173,"node_id":"MDU6TGFiZWw0NzgyNjExNzM=","url":"https://api.github.com/repos/Cloud-CV/EvalAI/labels/GSOC","name":"GSOC","color":"d93f0b","default":false},{"id":493103978,"node_id":"MDU6TGFiZWw0OTMxMDM5Nzg=","url":"https://api.github.com/repos/Cloud-CV/EvalAI/labels/backend","name":"backend","color":"1d76db","default":false},{"id":478261261,"node_id":"MDU6TGFiZWw0NzgyNjEyNjE=","url":"https://api.github.com/repos/Cloud-CV/EvalAI/labels/medium-difficulty","name":"medium-difficulty","color":"fbca04","default":false}],"state":"closed","locked":false,"assignee":{"login":"isht3","id":20072816,"node_id":"MDQ6VXNlcjIwMDcyODE2","avatar_url":"https://avatars1.githubusercontent.com/u/20072816?v=4","gravatar_id":"","url":"https://api.github.com/users/isht3","html_url":"https://github.com/isht3","followers_url":"https://api.github.com/users/isht3/followers","following_url":"https://api.github.com/users/isht3/following{/other_user}","gists_url":"https://api.github.com/users/isht3/gists{/gist_id}","starred_url":"https://api.github.com/users/isht3/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/isht3/subscriptions","organizations_url":"https://api.github.com/users/isht3/orgs","repos_url":"https://api.github.com/users/isht3/repos","events_url":"https://api.github.com/users/isht3/events{/privacy}","received_events_url":"https://api.github.com/users/isht3/received_events","type":"User","site_admin":false},"assignees":[{"login":"isht3","id":20072816,"node_id":"MDQ6VXNlcjIwMDcyODE2","avatar_url":"https://avatars1.githubusercontent.com/u/20072816?v=4","gravatar_id":"","url":"https://api.github.com/users/isht3","html_url":"https://github.com/isht3","followers_url":"https://api.github.com/users/isht3/followers","following_url":"https://api.github.com/users/isht3/following{/other_user}","gists_url":"https://api.github.com/users/isht3/gists{/gist_id}","starred_url":"https://api.github.com/users/isht3/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/isht3/subscriptions","organizations_url":"https://api.github.com/users/isht3/orgs","repos_url":"https://api.github.com/users/isht3/repos","events_url":"https://api.github.com/users/isht3/events{/privacy}","received_events_url":"https://api.github.com/users/isht3/received_events","type":"User","site_admin":false}],"milestone":null,"comments":19,"created_at":"2017-03-16T06:04:50Z","updated_at":"2018-08-19T04:11:20Z","closed_at":"2018-08-19T04:11:20Z","author_association":"MEMBER","body":"__Problem__\r\n\r\nWe need to propagate changes related to `test_annotation` and `evaluation_script` to the submission worker. The support for handling the change already in the worker is being taken care of in [Issue 840](https://github.com/Cloud-CV/EvalAI/issues/840). We need to add support for publishing message when ever any such change happens.\r\n\r\n__Please be more clear__ ?\r\n\r\nWhen ever `evaluation_script` or `test_annotation` file of Challenge or ChallengePhase changes, we need a way to tell this to submission worker that the files have changed and you need to update the files.\r\n\r\nThe best way can be by using a Queue based architecture and publishing a message to the exchange.\r\n\r\n__Why__ ?\r\n\r\nSubmission worker when starts loads all `evaluation_script` and `test_annotation` file in advance. In case if the Challenge Host updates any of the file, there is no way to communicate this to the worker.\r\n\r\nIf the worker is not communicated about the changes, then submissions may be evaluated with the wrong set of parameters.\r\n\r\n__Impact__ ?\r\n\r\nIt actually guarantees that whenever any submission is evaluated, it is evaluated against the write set of parameters.\r\n\r\n__How__ ?\r\n\r\nIt can be done by capturing the update of concerned models(here _Challenge_ and _ChallengePhase_).\r\n[Issue 839](https://github.com/Cloud-CV/EvalAI/issues/839) deals with this. The print statement in that issue can now be modified and there a new message can be published.\r\n\r\nThe __content__ of the message still needs to be figured out by the __contributor__. I will prefer that you publish only primary keys in the message but this is still open to discussion.\r\n\r\nRegarding how to publish message, you can take help from function `publish_submission_message` of `jobs/sender.py`. The published message will have a routing key of `challenge.edit.*`.\r\n\r\n__TakeAways__\r\n\r\nHow Exchanges work in RabbitMQ, how message are routed to the exchange ?\r\n\r\nAlso I will prefer that before coding, its better to discuss the approach over here only, so that we can get an idea and update if something is not right."},"comment":{"url":"https://api.github.com/repos/Cloud-CV/EvalAI/issues/comments/414102069","html_url":"https://github.com/Cloud-CV/EvalAI/issues/841#issuecomment-414102069","issue_url":"https://api.github.com/repos/Cloud-CV/EvalAI/issues/841","id":414102069,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNDEwMjA2OQ==","user":{"login":"deshraj","id":2945708,"node_id":"MDQ6VXNlcjI5NDU3MDg=","avatar_url":"https://avatars3.githubusercontent.com/u/2945708?v=4","gravatar_id":"","url":"https://api.github.com/users/deshraj","html_url":"https://github.com/deshraj","followers_url":"https://api.github.com/users/deshraj/followers","following_url":"https://api.github.com/users/deshraj/following{/other_user}","gists_url":"https://api.github.com/users/deshraj/gists{/gist_id}","starred_url":"https://api.github.com/users/deshraj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deshraj/subscriptions","organizations_url":"https://api.github.com/users/deshraj/orgs","repos_url":"https://api.github.com/users/deshraj/repos","events_url":"https://api.github.com/users/deshraj/events{/privacy}","received_events_url":"https://api.github.com/users/deshraj/received_events","type":"User","site_admin":false},"created_at":"2018-08-19T04:11:20Z","updated_at":"2018-08-19T04:11:20Z","author_association":"MEMBER","body":"Similar to #842. Closing it. Thanks for pointing out @taranjeet :) "}}
{"actor":{"display_login":"deshraj"}}