{"action":"created","issue":{"url":"https://api.github.com/repos/open-contracting/ocdsdata/issues/100","repository_url":"https://api.github.com/repos/open-contracting/ocdsdata","labels_url":"https://api.github.com/repos/open-contracting/ocdsdata/issues/100/labels{/name}","comments_url":"https://api.github.com/repos/open-contracting/ocdsdata/issues/100/comments","events_url":"https://api.github.com/repos/open-contracting/ocdsdata/issues/100/events","html_url":"https://github.com/open-contracting/ocdsdata/issues/100","id":330666280,"node_id":"MDU6SXNzdWUzMzA2NjYyODA=","number":100,"title":"Test & bugfix scrapers","user":{"login":"robredpath","id":641009,"node_id":"MDQ6VXNlcjY0MTAwOQ==","avatar_url":"https://avatars3.githubusercontent.com/u/641009?v=4","gravatar_id":"","url":"https://api.github.com/users/robredpath","html_url":"https://github.com/robredpath","followers_url":"https://api.github.com/users/robredpath/followers","following_url":"https://api.github.com/users/robredpath/following{/other_user}","gists_url":"https://api.github.com/users/robredpath/gists{/gist_id}","starred_url":"https://api.github.com/users/robredpath/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robredpath/subscriptions","organizations_url":"https://api.github.com/users/robredpath/orgs","repos_url":"https://api.github.com/users/robredpath/repos","events_url":"https://api.github.com/users/robredpath/events{/privacy}","received_events_url":"https://api.github.com/users/robredpath/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"robredpath","id":641009,"node_id":"MDQ6VXNlcjY0MTAwOQ==","avatar_url":"https://avatars3.githubusercontent.com/u/641009?v=4","gravatar_id":"","url":"https://api.github.com/users/robredpath","html_url":"https://github.com/robredpath","followers_url":"https://api.github.com/users/robredpath/followers","following_url":"https://api.github.com/users/robredpath/following{/other_user}","gists_url":"https://api.github.com/users/robredpath/gists{/gist_id}","starred_url":"https://api.github.com/users/robredpath/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robredpath/subscriptions","organizations_url":"https://api.github.com/users/robredpath/orgs","repos_url":"https://api.github.com/users/robredpath/repos","events_url":"https://api.github.com/users/robredpath/events{/privacy}","received_events_url":"https://api.github.com/users/robredpath/received_events","type":"User","site_admin":false},"assignees":[{"login":"robredpath","id":641009,"node_id":"MDQ6VXNlcjY0MTAwOQ==","avatar_url":"https://avatars3.githubusercontent.com/u/641009?v=4","gravatar_id":"","url":"https://api.github.com/users/robredpath","html_url":"https://github.com/robredpath","followers_url":"https://api.github.com/users/robredpath/followers","following_url":"https://api.github.com/users/robredpath/following{/other_user}","gists_url":"https://api.github.com/users/robredpath/gists{/gist_id}","starred_url":"https://api.github.com/users/robredpath/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robredpath/subscriptions","organizations_url":"https://api.github.com/users/robredpath/orgs","repos_url":"https://api.github.com/users/robredpath/repos","events_url":"https://api.github.com/users/robredpath/events{/privacy}","received_events_url":"https://api.github.com/users/robredpath/received_events","type":"User","site_admin":false},{"login":"odscjames","id":37832009,"node_id":"MDQ6VXNlcjM3ODMyMDA5","avatar_url":"https://avatars2.githubusercontent.com/u/37832009?v=4","gravatar_id":"","url":"https://api.github.com/users/odscjames","html_url":"https://github.com/odscjames","followers_url":"https://api.github.com/users/odscjames/followers","following_url":"https://api.github.com/users/odscjames/following{/other_user}","gists_url":"https://api.github.com/users/odscjames/gists{/gist_id}","starred_url":"https://api.github.com/users/odscjames/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/odscjames/subscriptions","organizations_url":"https://api.github.com/users/odscjames/orgs","repos_url":"https://api.github.com/users/odscjames/repos","events_url":"https://api.github.com/users/odscjames/events{/privacy}","received_events_url":"https://api.github.com/users/odscjames/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-06-08T13:50:08Z","updated_at":"2018-06-14T10:56:14Z","closed_at":null,"author_association":"MEMBER","body":"- [ ] armenia.py\r\n- [ ] australia.py\r\n- [ ] canada_buyandsell.py\r\n- [ ] canada_montreal.py\r\n- [ ] colombia.py\r\n- [ ] mexico_administracion_publica_federal.py\r\n- [ ] mexico_cdmx.py\r\n- [ ] mexico_grupo_aeroporto.py\r\n- [ ] mexico_inai.py\r\n- [ ] mexico_jalisco.py\r\n- [ ] moldova.py\r\n- [ ] paraguay_dncp.py\r\n- [ ] paraguay_hacienda.py\r\n- [ ] taiwan.py\r\n- [ ] uganda.py\r\n- [ ] uk_contracts_finder.py #103\r\n- [ ] ukraine.py #102\r\n- [ ] uruguay.py\r\n\r\nTick off above when checked or when broken out to a separate ticket (if we realise one is having big problems let's make it a separate ticket and try and keep this one relatively clean - they will probably be separate pull requests anyway!)\r\n"},"comment":{"url":"https://api.github.com/repos/open-contracting/ocdsdata/issues/comments/397254701","html_url":"https://github.com/open-contracting/ocdsdata/issues/100#issuecomment-397254701","issue_url":"https://api.github.com/repos/open-contracting/ocdsdata/issues/100","id":397254701,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NzI1NDcwMQ==","user":{"login":"odscjames","id":37832009,"node_id":"MDQ6VXNlcjM3ODMyMDA5","avatar_url":"https://avatars2.githubusercontent.com/u/37832009?v=4","gravatar_id":"","url":"https://api.github.com/users/odscjames","html_url":"https://github.com/odscjames","followers_url":"https://api.github.com/users/odscjames/followers","following_url":"https://api.github.com/users/odscjames/following{/other_user}","gists_url":"https://api.github.com/users/odscjames/gists{/gist_id}","starred_url":"https://api.github.com/users/odscjames/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/odscjames/subscriptions","organizations_url":"https://api.github.com/users/odscjames/orgs","repos_url":"https://api.github.com/users/odscjames/repos","events_url":"https://api.github.com/users/odscjames/events{/privacy}","received_events_url":"https://api.github.com/users/odscjames/received_events","type":"User","site_admin":false},"created_at":"2018-06-14T10:56:14Z","updated_at":"2018-06-14T10:56:14Z","author_association":"CONTRIBUTOR","body":"#108 should contain a useful function in the DB class we can reuse but here's my first quick try at some SQL .... get the ID of the source_session row you want to delete. In one transaction run:\r\n\r\n````\r\ndelete from release_check where release_id in (select release.id from release join source_session_file_status on source_session_file_status.id =  release.source_session_file_status_id where source_session_file_status.source_session_id = $id);\r\n\r\ndelete from record_check where record_id in (select record.id from record join source_session_file_status on source_session_file_status.id =  record.source_session_file_status_id where source_session_file_status.source_session_id = $id);\r\n\r\ndelete from record where source_session_file_status_id in (select id from source_session_file_status where source_session_id = $id);\r\n\r\ndelete from release where source_session_file_status_id in (select id from source_session_file_status where source_session_id = $id);\r\n\r\ndelete from source_session_file_status where source_session_id = $id;\r\n\r\ndelete from source_session where id = $id;\r\n````\r\n\r\nThen in one transaction per statement run:\r\n\r\n````\r\ndelete from data where id not in (select data_id from release union select data_id from record);\r\n\r\ndelete from package_data where id not in (select package_data_id from release union select package_data_id from record);\r\n````\r\n\r\nNot sure how this will work at scale yet but we'll start by assuming the DB is better than what we could do and only look at optimizations later if we need to. We'll properly test and play as we do other work."}}
{"actor":{"display_login":"odscjames"}}