{"action":"created","issue":{"url":"https://api.github.com/repos/PanDAWMS/dkb/issues/93","repository_url":"https://api.github.com/repos/PanDAWMS/dkb","labels_url":"https://api.github.com/repos/PanDAWMS/dkb/issues/93/labels{/name}","comments_url":"https://api.github.com/repos/PanDAWMS/dkb/issues/93/comments","events_url":"https://api.github.com/repos/PanDAWMS/dkb/issues/93/events","html_url":"https://github.com/PanDAWMS/dkb/pull/93","id":280704268,"number":93,"title":"Stage 009 (Oracle Connector): update","user":{"login":"mgolosova","id":10220547,"avatar_url":"https://avatars1.githubusercontent.com/u/10220547?v=4","gravatar_id":"","url":"https://api.github.com/users/mgolosova","html_url":"https://github.com/mgolosova","followers_url":"https://api.github.com/users/mgolosova/followers","following_url":"https://api.github.com/users/mgolosova/following{/other_user}","gists_url":"https://api.github.com/users/mgolosova/gists{/gist_id}","starred_url":"https://api.github.com/users/mgolosova/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgolosova/subscriptions","organizations_url":"https://api.github.com/users/mgolosova/orgs","repos_url":"https://api.github.com/users/mgolosova/repos","events_url":"https://api.github.com/users/mgolosova/events{/privacy}","received_events_url":"https://api.github.com/users/mgolosova/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"mgolosova","id":10220547,"avatar_url":"https://avatars1.githubusercontent.com/u/10220547?v=4","gravatar_id":"","url":"https://api.github.com/users/mgolosova","html_url":"https://github.com/mgolosova","followers_url":"https://api.github.com/users/mgolosova/followers","following_url":"https://api.github.com/users/mgolosova/following{/other_user}","gists_url":"https://api.github.com/users/mgolosova/gists{/gist_id}","starred_url":"https://api.github.com/users/mgolosova/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgolosova/subscriptions","organizations_url":"https://api.github.com/users/mgolosova/orgs","repos_url":"https://api.github.com/users/mgolosova/repos","events_url":"https://api.github.com/users/mgolosova/events{/privacy}","received_events_url":"https://api.github.com/users/mgolosova/received_events","type":"User","site_admin":false},"assignees":[{"login":"mgolosova","id":10220547,"avatar_url":"https://avatars1.githubusercontent.com/u/10220547?v=4","gravatar_id":"","url":"https://api.github.com/users/mgolosova","html_url":"https://github.com/mgolosova","followers_url":"https://api.github.com/users/mgolosova/followers","following_url":"https://api.github.com/users/mgolosova/following{/other_user}","gists_url":"https://api.github.com/users/mgolosova/gists{/gist_id}","starred_url":"https://api.github.com/users/mgolosova/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgolosova/subscriptions","organizations_url":"https://api.github.com/users/mgolosova/orgs","repos_url":"https://api.github.com/users/mgolosova/repos","events_url":"https://api.github.com/users/mgolosova/events{/privacy}","received_events_url":"https://api.github.com/users/mgolosova/received_events","type":"User","site_admin":false}],"milestone":null,"comments":8,"created_at":"2017-12-09T09:29:20Z","updated_at":"2018-02-05T08:12:48Z","closed_at":null,"author_association":"COLLABORATOR","pull_request":{"url":"https://api.github.com/repos/PanDAWMS/dkb/pulls/93","html_url":"https://github.com/PanDAWMS/dkb/pull/93","diff_url":"https://github.com/PanDAWMS/dkb/pull/93.diff","patch_url":"https://github.com/PanDAWMS/dkb/pull/93.patch"},"body":"Improvements and fixes.\r\n\r\nMajor changes:\r\n* current offset is to be stored in a separate file (instead of main config) [1];\r\n* `squash_records()` function was re-implemented so that it work as a generator [2];\r\n* query files configuration is considered to be related to config dir;\r\n* output in an unbuffered mode;\r\n* issue #92 fixed (restoring of broken session/connection to Oracle);\r\n\r\nMinor changes:\r\n* docstrings added/updated;\r\n* internal format for datetime became `datetime.datetime` (instead of `str`);\r\n* parameter name changed for some functions;\r\n* more universal algorithm for `squash_records()`;\r\n* separate class for `dbConnection`;\r\n* help message fixed.\r\n\r\nToDo:\r\n- [ ] issue #92 *(need to check it again, it seems to me that it didn`t work in the \"production\")*\r\n- [x] reset current offset if `initial` one is later\r\n\r\n---\r\n[1] It is required for automatic deployment via Puppet:\r\n* the config file is to be generated automatically by Puppet agent running on the node, so on every run it is to be rewritten;\r\n* we would like to restart the `data4es` chain if the config file has changed (say, Oracle access parameters or SQL queries changed). If we store the `current_offset` in the same file, it will be considered \"changed\" on every Puppet agent run.\r\n\r\n[2] We try to avoid storing all the records received from Oracle in the memory. Instead we use generators to process one record at a time. The `squash_records` appeared the only function left as non-generator, so despite other functions supposed to operate with one record at a time, all the records from `datasets` query inevitably massed in the memory as soon as `squash_records` was called."},"comment":{"url":"https://api.github.com/repos/PanDAWMS/dkb/issues/comments/363010037","html_url":"https://github.com/PanDAWMS/dkb/pull/93#issuecomment-363010037","issue_url":"https://api.github.com/repos/PanDAWMS/dkb/issues/93","id":363010037,"user":{"login":"mgolosova","id":10220547,"avatar_url":"https://avatars1.githubusercontent.com/u/10220547?v=4","gravatar_id":"","url":"https://api.github.com/users/mgolosova","html_url":"https://github.com/mgolosova","followers_url":"https://api.github.com/users/mgolosova/followers","following_url":"https://api.github.com/users/mgolosova/following{/other_user}","gists_url":"https://api.github.com/users/mgolosova/gists{/gist_id}","starred_url":"https://api.github.com/users/mgolosova/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgolosova/subscriptions","organizations_url":"https://api.github.com/users/mgolosova/orgs","repos_url":"https://api.github.com/users/mgolosova/repos","events_url":"https://api.github.com/users/mgolosova/events{/privacy}","received_events_url":"https://api.github.com/users/mgolosova/received_events","type":"User","site_admin":false},"created_at":"2018-02-05T08:12:48Z","updated_at":"2018-02-05T08:12:48Z","author_association":"COLLABORATOR","body":"Tested Issue #92 in case of long timeout between two SQL requests to Oracle, and it worked fine.\r\n\r\nStill I am not sure that \"that`s it\" and think I might have missed something...\r\nGoing to test PR #84 (AMI part of the dataflow) together with this PR (as these changes are vital for the proper work of the whole process), and if I see any sights of `ORA-02396`, I will update information here."}}
{"actor":{"display_login":"mgolosova"}}