{"action":"closed","number":2563,"pull_request":{"url":"https://api.github.com/repos/timgrossmann/InstaPy/pulls/2563","id":204634094,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA0NjM0MDk0","html_url":"https://github.com/timgrossmann/InstaPy/pull/2563","diff_url":"https://github.com/timgrossmann/InstaPy/pull/2563.diff","patch_url":"https://github.com/timgrossmann/InstaPy/pull/2563.patch","issue_url":"https://api.github.com/repos/timgrossmann/InstaPy/issues/2563","number":2563,"state":"closed","locked":false,"title":"Dedicated Data Base to provide robust 'record activity' and 'follow restriction' and support future DB operations easily","user":{"login":"uluQulu","id":33269249,"node_id":"MDQ6VXNlcjMzMjY5MjQ5","avatar_url":"https://avatars3.githubusercontent.com/u/33269249?v=4","gravatar_id":"","url":"https://api.github.com/users/uluQulu","html_url":"https://github.com/uluQulu","followers_url":"https://api.github.com/users/uluQulu/followers","following_url":"https://api.github.com/users/uluQulu/following{/other_user}","gists_url":"https://api.github.com/users/uluQulu/gists{/gist_id}","starred_url":"https://api.github.com/users/uluQulu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uluQulu/subscriptions","organizations_url":"https://api.github.com/users/uluQulu/orgs","repos_url":"https://api.github.com/users/uluQulu/repos","events_url":"https://api.github.com/users/uluQulu/events{/privacy}","received_events_url":"https://api.github.com/users/uluQulu/received_events","type":"User","site_admin":false},"body":"Hey there! 🙂\r\n\r\n#### The work I talked about has been written and I introduce the new set of inner tools and also the agreement about the bonds of follow restriction has also been accomplished.  \r\n\r\n_I will talk about the major changes coming with this PR._\r\n\r\n**1-)** Now there is a database engine which is making a fresh new database at request\r\n+ **previously**\r\ninstapy.db (_came with the repo_)\r\n____________________________________________________________________________________________________________\r\n|           _TABLES_           |                                                 _COLUMNS_                                                        |\r\n| **statistics**                |  `likes`, `comments`, `follows`, `unfollows`, `server_calls`, `created`                      |\r\n_____________________________________________________________________________________________________________\r\n+ **now**\r\ninstapy.db (_written at request_)\r\n____________________________________________________________________________________________________________\r\n|           _TABLES_          |                                                 _COLUMNS_                                                        |\r\n| **profiles**                 |  `id`, `name`                                                                                                         |\r\n| **recordActivity**       |  `profile_id`, `likes`, `comments`, `follows`, `unfollows`, `server_calls`, `created`  |\r\n| **followRestriction**  |  `profile_id`, `username`, `times`                                                                          |\r\n____________________________________________________________________________________________________________\r\n\r\n**2-)** Priorly called **instapy.db** file held `statistics` table which is now renamed to `recordActivity` and the content of that table is the same, except a new column- `profile_id`.\r\n\r\n**3-)** There is only 1 database- **instapy.db** holding three tables `profiles`, `recordActivity`, `followRestriction`. Please see the **database_engine.py** file to see their column types and more.\r\n\r\n**4-)** `profiles` table is holding the names of the unique users (_\"`owner`\" users- the users who started a **quickstart** script_) and their `id` is being **referenced** by other tables. This kind of referencing saves up tons of space , provide clearer structure and powerful functionalities and some  speed.\r\n\r\n**5-)** The database engine is written to handle every worse situation.  e.g.,\r\n+ you can remove a database at the time program running, there, it will make a database again\r\n+ you can give a custom location of database using Settings class and there if the location does not contain a database, then it will make a new database\r\n+ etc.\r\n\r\n**6-)** Now follow restriction is fully handled by the realtime database data which is being written one by one and has no any data leak.\r\n\r\n**7-)** To provide speed in follow restriction querying, I used a good `WHERE` clause which is fetching only for the _username_ rather than fetching whole data and comparing it. Speed is so cool here.\r\n\r\n**8-)** I renamed `statistics` to `recordActivity` cos due to smart naming policy.  \r\n\r\n**9-)** Now, follow restriction and record activity data is also being dumped to local files in logs folder for the user to view them easily.\r\n+ I used the structure based on the username to solve the possible misbehaviours during multi logs turned off. That's why the data will be dumped at the username level.  \r\n+ Note that those JSONs are for viewing and has no any internal usage.\r\n\r\n**10-)** Right, having considered that the users might have existing data in previous **followRestriction.json** file, I made a design choice. The thing is while it dumps new data to _follow restriction_ and/or _record activity_ local JSONs files, it will preserve the existing data in there 🤝🏼.\r\n* E.g. you have 2 elements in **followRestriction.json** before merging this PR,\r\n```javascript\r\n{\"old_user1\":3, \"old_user2\":1}\r\n```\r\n* then you merge this PR **and** start/finish a good session, then that file will be like,\r\n```javascript\r\n{\"old_user1\":3, \"old_user2\":1, \"YOURUSERNAME\":{\"new_user1\":1, \"new_user2\":4, \"new_user3\":2}}\r\n```\r\n* _so that you can put that **old** data into correct username label or just remove it._\r\n>If it gets hard to understand, start using it you will see how it works.\r\n\r\n**11-)** Database tables' integrity will be checked at the session start and once the database file does not exist. So, the only worse situation can be, you drop the _old structured_ **instapy.db** file inside that location once the program is running, but I think of somebody doing it is crazy 😂  \r\nThe other times, it will just pass back the address of the database in use.\r\n\r\n**12-)** For speed, I used simple data base write model- _a row at a time_ for writing follow restriction data.\r\nIt is fast for both querying and writing data.  \r\n\r\n**13-)** Removed the default loaded instapy.db file from the repo cos of it is gonna written from scratch at database request. As a result, users will no more have to stash the **modified** instapy.db file while pulling new commits. Cos now, **instapy.db** file will be an **unstaged** file 🙂.  \r\n\r\n**14-)** Matters I forgot to say! 😀 Please see the changes and ask for unclear lines in questions.\r\n\r\n\r\n\r\n#### There are also other changes in several different parts came with this PR:\r\n+ using **sorted** algorithm to remove duplicates faster at `get_links_for_username`\r\n+ solve the case of empty strings while getting followers and following counts\r\n+ fix follow button in `follow_given_user` definition to work properly in `Follow Back` buttons, too\r\n+ some other slight changes for performance and good looks\r\n\r\n\r\n#### **PROS**:\r\n+ Previously dumping json files which held the very important data of follow restriction was problematic and once that JSON was getting huge, it was failing to dump in some session exit and your whole previous data was lost.  I don't even talk about corruption. \r\nBut now, with this dedicated DB structure, it will always keep a robust data in there.\r\n+ With this Data Base model, new tables or databases can be written **easily** and be served to required functions.\r\n\r\n#### **CONS**:\r\nIn future there have to be written a new module to remove **so old** records in the database (_e.g. 6 months old data_), vice verse it is obviously gonna cost an inefficiency in querying time.\r\n\r\n\r\n#### LAST NOTES\r\nA casual user will not notice any of these changes cos they are intended mostly for  inner use.\r\nExcept, now there will be a noticeably new file- **recordActivity.json** holding record activity data in logs folder which will be a very useful file for everybody.\r\n_I want to thank @converge once more for his amazing DB work in here which held the light on the road preparing this._\r\n\r\nWriting this tool was easy but the phase of deciding the design was the most time consuming but hopefully I found all solutions for it at its best and wish it is gonna be useful to you! \r\n\r\n\r\nCheers 😁","created_at":"2018-07-29T09:45:07Z","updated_at":"2018-07-30T10:50:58Z","closed_at":"2018-07-30T10:50:58Z","merged_at":"2018-07-30T10:50:58Z","merge_commit_sha":"c87b64506576492ea5a8b021c1b4f78cfc3083b0","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/timgrossmann/InstaPy/pulls/2563/commits","review_comments_url":"https://api.github.com/repos/timgrossmann/InstaPy/pulls/2563/comments","review_comment_url":"https://api.github.com/repos/timgrossmann/InstaPy/pulls/comments{/number}","comments_url":"https://api.github.com/repos/timgrossmann/InstaPy/issues/2563/comments","statuses_url":"https://api.github.com/repos/timgrossmann/InstaPy/statuses/6099336932ec647d4454e7ede6ff599403326d54","head":{"label":"uluQulu:master","ref":"master","sha":"6099336932ec647d4454e7ede6ff599403326d54","user":{"login":"uluQulu","id":33269249,"node_id":"MDQ6VXNlcjMzMjY5MjQ5","avatar_url":"https://avatars3.githubusercontent.com/u/33269249?v=4","gravatar_id":"","url":"https://api.github.com/users/uluQulu","html_url":"https://github.com/uluQulu","followers_url":"https://api.github.com/users/uluQulu/followers","following_url":"https://api.github.com/users/uluQulu/following{/other_user}","gists_url":"https://api.github.com/users/uluQulu/gists{/gist_id}","starred_url":"https://api.github.com/users/uluQulu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uluQulu/subscriptions","organizations_url":"https://api.github.com/users/uluQulu/orgs","repos_url":"https://api.github.com/users/uluQulu/repos","events_url":"https://api.github.com/users/uluQulu/events{/privacy}","received_events_url":"https://api.github.com/users/uluQulu/received_events","type":"User","site_admin":false},"repo":{"id":142708271,"node_id":"MDEwOlJlcG9zaXRvcnkxNDI3MDgyNzE=","name":"InstaPy","full_name":"uluQulu/InstaPy","owner":{"login":"uluQulu","id":33269249,"node_id":"MDQ6VXNlcjMzMjY5MjQ5","avatar_url":"https://avatars3.githubusercontent.com/u/33269249?v=4","gravatar_id":"","url":"https://api.github.com/users/uluQulu","html_url":"https://github.com/uluQulu","followers_url":"https://api.github.com/users/uluQulu/followers","following_url":"https://api.github.com/users/uluQulu/following{/other_user}","gists_url":"https://api.github.com/users/uluQulu/gists{/gist_id}","starred_url":"https://api.github.com/users/uluQulu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uluQulu/subscriptions","organizations_url":"https://api.github.com/users/uluQulu/orgs","repos_url":"https://api.github.com/users/uluQulu/repos","events_url":"https://api.github.com/users/uluQulu/events{/privacy}","received_events_url":"https://api.github.com/users/uluQulu/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/uluQulu/InstaPy","description":"📷 Instagram Bot - Like/Comment/Follow Automation Script","fork":true,"url":"https://api.github.com/repos/uluQulu/InstaPy","forks_url":"https://api.github.com/repos/uluQulu/InstaPy/forks","keys_url":"https://api.github.com/repos/uluQulu/InstaPy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/uluQulu/InstaPy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/uluQulu/InstaPy/teams","hooks_url":"https://api.github.com/repos/uluQulu/InstaPy/hooks","issue_events_url":"https://api.github.com/repos/uluQulu/InstaPy/issues/events{/number}","events_url":"https://api.github.com/repos/uluQulu/InstaPy/events","assignees_url":"https://api.github.com/repos/uluQulu/InstaPy/assignees{/user}","branches_url":"https://api.github.com/repos/uluQulu/InstaPy/branches{/branch}","tags_url":"https://api.github.com/repos/uluQulu/InstaPy/tags","blobs_url":"https://api.github.com/repos/uluQulu/InstaPy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/uluQulu/InstaPy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/uluQulu/InstaPy/git/refs{/sha}","trees_url":"https://api.github.com/repos/uluQulu/InstaPy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/uluQulu/InstaPy/statuses/{sha}","languages_url":"https://api.github.com/repos/uluQulu/InstaPy/languages","stargazers_url":"https://api.github.com/repos/uluQulu/InstaPy/stargazers","contributors_url":"https://api.github.com/repos/uluQulu/InstaPy/contributors","subscribers_url":"https://api.github.com/repos/uluQulu/InstaPy/subscribers","subscription_url":"https://api.github.com/repos/uluQulu/InstaPy/subscription","commits_url":"https://api.github.com/repos/uluQulu/InstaPy/commits{/sha}","git_commits_url":"https://api.github.com/repos/uluQulu/InstaPy/git/commits{/sha}","comments_url":"https://api.github.com/repos/uluQulu/InstaPy/comments{/number}","issue_comment_url":"https://api.github.com/repos/uluQulu/InstaPy/issues/comments{/number}","contents_url":"https://api.github.com/repos/uluQulu/InstaPy/contents/{+path}","compare_url":"https://api.github.com/repos/uluQulu/InstaPy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/uluQulu/InstaPy/merges","archive_url":"https://api.github.com/repos/uluQulu/InstaPy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/uluQulu/InstaPy/downloads","issues_url":"https://api.github.com/repos/uluQulu/InstaPy/issues{/number}","pulls_url":"https://api.github.com/repos/uluQulu/InstaPy/pulls{/number}","milestones_url":"https://api.github.com/repos/uluQulu/InstaPy/milestones{/number}","notifications_url":"https://api.github.com/repos/uluQulu/InstaPy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/uluQulu/InstaPy/labels{/name}","releases_url":"https://api.github.com/repos/uluQulu/InstaPy/releases{/id}","deployments_url":"https://api.github.com/repos/uluQulu/InstaPy/deployments","created_at":"2018-07-28T20:32:46Z","updated_at":"2018-07-29T10:22:32Z","pushed_at":"2018-07-29T10:22:30Z","git_url":"git://github.com/uluQulu/InstaPy.git","ssh_url":"git@github.com:uluQulu/InstaPy.git","clone_url":"https://github.com/uluQulu/InstaPy.git","svn_url":"https://github.com/uluQulu/InstaPy","homepage":"https://timgrossmann.github.io/InstaPy/","size":78088,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit","node_id":"MDc6TGljZW5zZTEz"},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"timgrossmann:master","ref":"master","sha":"c63be3a19f0858b189fd9eee165fdf81ad42583a","user":{"login":"timgrossmann","id":16529337,"node_id":"MDQ6VXNlcjE2NTI5MzM3","avatar_url":"https://avatars2.githubusercontent.com/u/16529337?v=4","gravatar_id":"","url":"https://api.github.com/users/timgrossmann","html_url":"https://github.com/timgrossmann","followers_url":"https://api.github.com/users/timgrossmann/followers","following_url":"https://api.github.com/users/timgrossmann/following{/other_user}","gists_url":"https://api.github.com/users/timgrossmann/gists{/gist_id}","starred_url":"https://api.github.com/users/timgrossmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timgrossmann/subscriptions","organizations_url":"https://api.github.com/users/timgrossmann/orgs","repos_url":"https://api.github.com/users/timgrossmann/repos","events_url":"https://api.github.com/users/timgrossmann/events{/privacy}","received_events_url":"https://api.github.com/users/timgrossmann/received_events","type":"User","site_admin":false},"repo":{"id":69290591,"node_id":"MDEwOlJlcG9zaXRvcnk2OTI5MDU5MQ==","name":"InstaPy","full_name":"timgrossmann/InstaPy","owner":{"login":"timgrossmann","id":16529337,"node_id":"MDQ6VXNlcjE2NTI5MzM3","avatar_url":"https://avatars2.githubusercontent.com/u/16529337?v=4","gravatar_id":"","url":"https://api.github.com/users/timgrossmann","html_url":"https://github.com/timgrossmann","followers_url":"https://api.github.com/users/timgrossmann/followers","following_url":"https://api.github.com/users/timgrossmann/following{/other_user}","gists_url":"https://api.github.com/users/timgrossmann/gists{/gist_id}","starred_url":"https://api.github.com/users/timgrossmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timgrossmann/subscriptions","organizations_url":"https://api.github.com/users/timgrossmann/orgs","repos_url":"https://api.github.com/users/timgrossmann/repos","events_url":"https://api.github.com/users/timgrossmann/events{/privacy}","received_events_url":"https://api.github.com/users/timgrossmann/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/timgrossmann/InstaPy","description":"📷 Instagram Bot - Like/Comment/Follow Automation Script","fork":false,"url":"https://api.github.com/repos/timgrossmann/InstaPy","forks_url":"https://api.github.com/repos/timgrossmann/InstaPy/forks","keys_url":"https://api.github.com/repos/timgrossmann/InstaPy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/timgrossmann/InstaPy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/timgrossmann/InstaPy/teams","hooks_url":"https://api.github.com/repos/timgrossmann/InstaPy/hooks","issue_events_url":"https://api.github.com/repos/timgrossmann/InstaPy/issues/events{/number}","events_url":"https://api.github.com/repos/timgrossmann/InstaPy/events","assignees_url":"https://api.github.com/repos/timgrossmann/InstaPy/assignees{/user}","branches_url":"https://api.github.com/repos/timgrossmann/InstaPy/branches{/branch}","tags_url":"https://api.github.com/repos/timgrossmann/InstaPy/tags","blobs_url":"https://api.github.com/repos/timgrossmann/InstaPy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/timgrossmann/InstaPy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/timgrossmann/InstaPy/git/refs{/sha}","trees_url":"https://api.github.com/repos/timgrossmann/InstaPy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/timgrossmann/InstaPy/statuses/{sha}","languages_url":"https://api.github.com/repos/timgrossmann/InstaPy/languages","stargazers_url":"https://api.github.com/repos/timgrossmann/InstaPy/stargazers","contributors_url":"https://api.github.com/repos/timgrossmann/InstaPy/contributors","subscribers_url":"https://api.github.com/repos/timgrossmann/InstaPy/subscribers","subscription_url":"https://api.github.com/repos/timgrossmann/InstaPy/subscription","commits_url":"https://api.github.com/repos/timgrossmann/InstaPy/commits{/sha}","git_commits_url":"https://api.github.com/repos/timgrossmann/InstaPy/git/commits{/sha}","comments_url":"https://api.github.com/repos/timgrossmann/InstaPy/comments{/number}","issue_comment_url":"https://api.github.com/repos/timgrossmann/InstaPy/issues/comments{/number}","contents_url":"https://api.github.com/repos/timgrossmann/InstaPy/contents/{+path}","compare_url":"https://api.github.com/repos/timgrossmann/InstaPy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/timgrossmann/InstaPy/merges","archive_url":"https://api.github.com/repos/timgrossmann/InstaPy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/timgrossmann/InstaPy/downloads","issues_url":"https://api.github.com/repos/timgrossmann/InstaPy/issues{/number}","pulls_url":"https://api.github.com/repos/timgrossmann/InstaPy/pulls{/number}","milestones_url":"https://api.github.com/repos/timgrossmann/InstaPy/milestones{/number}","notifications_url":"https://api.github.com/repos/timgrossmann/InstaPy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/timgrossmann/InstaPy/labels{/name}","releases_url":"https://api.github.com/repos/timgrossmann/InstaPy/releases{/id}","deployments_url":"https://api.github.com/repos/timgrossmann/InstaPy/deployments","created_at":"2016-09-26T20:36:26Z","updated_at":"2018-07-30T10:31:22Z","pushed_at":"2018-07-30T10:50:58Z","git_url":"git://github.com/timgrossmann/InstaPy.git","ssh_url":"git@github.com:timgrossmann/InstaPy.git","clone_url":"https://github.com/timgrossmann/InstaPy.git","svn_url":"https://github.com/timgrossmann/InstaPy","homepage":"https://timgrossmann.github.io/InstaPy/","size":78093,"stargazers_count":5055,"watchers_count":5055,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":1307,"mirror_url":null,"archived":false,"open_issues_count":1086,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit","node_id":"MDc6TGljZW5zZTEz"},"forks":1307,"open_issues":1086,"watchers":5055,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/timgrossmann/InstaPy/pulls/2563"},"html":{"href":"https://github.com/timgrossmann/InstaPy/pull/2563"},"issue":{"href":"https://api.github.com/repos/timgrossmann/InstaPy/issues/2563"},"comments":{"href":"https://api.github.com/repos/timgrossmann/InstaPy/issues/2563/comments"},"review_comments":{"href":"https://api.github.com/repos/timgrossmann/InstaPy/pulls/2563/comments"},"review_comment":{"href":"https://api.github.com/repos/timgrossmann/InstaPy/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/timgrossmann/InstaPy/pulls/2563/commits"},"statuses":{"href":"https://api.github.com/repos/timgrossmann/InstaPy/statuses/6099336932ec647d4454e7ede6ff599403326d54"}},"author_association":"COLLABORATOR","merged":true,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":{"login":"timgrossmann","id":16529337,"node_id":"MDQ6VXNlcjE2NTI5MzM3","avatar_url":"https://avatars2.githubusercontent.com/u/16529337?v=4","gravatar_id":"","url":"https://api.github.com/users/timgrossmann","html_url":"https://github.com/timgrossmann","followers_url":"https://api.github.com/users/timgrossmann/followers","following_url":"https://api.github.com/users/timgrossmann/following{/other_user}","gists_url":"https://api.github.com/users/timgrossmann/gists{/gist_id}","starred_url":"https://api.github.com/users/timgrossmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timgrossmann/subscriptions","organizations_url":"https://api.github.com/users/timgrossmann/orgs","repos_url":"https://api.github.com/users/timgrossmann/repos","events_url":"https://api.github.com/users/timgrossmann/events{/privacy}","received_events_url":"https://api.github.com/users/timgrossmann/received_events","type":"User","site_admin":false},"comments":1,"review_comments":0,"maintainer_can_modify":false,"commits":1,"additions":401,"deletions":122,"changed_files":7}}
{"actor":{"display_login":"timgrossmann"}}