{"action":"created","issue":{"url":"https://api.github.com/repos/HackEdit/hackedit/issues/26","repository_url":"https://api.github.com/repos/HackEdit/hackedit","labels_url":"https://api.github.com/repos/HackEdit/hackedit/issues/26/labels{/name}","comments_url":"https://api.github.com/repos/HackEdit/hackedit/issues/26/comments","events_url":"https://api.github.com/repos/HackEdit/hackedit/issues/26/events","html_url":"https://github.com/HackEdit/hackedit/issues/26","id":116813092,"number":26,"title":"Indexing overhaul","user":{"login":"ColinDuquesnoy","id":1681217,"avatar_url":"https://avatars.githubusercontent.com/u/1681217?v=3","gravatar_id":"","url":"https://api.github.com/users/ColinDuquesnoy","html_url":"https://github.com/ColinDuquesnoy","followers_url":"https://api.github.com/users/ColinDuquesnoy/followers","following_url":"https://api.github.com/users/ColinDuquesnoy/following{/other_user}","gists_url":"https://api.github.com/users/ColinDuquesnoy/gists{/gist_id}","starred_url":"https://api.github.com/users/ColinDuquesnoy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ColinDuquesnoy/subscriptions","organizations_url":"https://api.github.com/users/ColinDuquesnoy/orgs","repos_url":"https://api.github.com/users/ColinDuquesnoy/repos","events_url":"https://api.github.com/users/ColinDuquesnoy/events{/privacy}","received_events_url":"https://api.github.com/users/ColinDuquesnoy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/HackEdit/hackedit/labels/enhancement","name":"enhancement","color":"009800"},{"url":"https://api.github.com/repos/HackEdit/hackedit/labels/hackedit","name":"hackedit","color":"fef2c0"},{"url":"https://api.github.com/repos/HackEdit/hackedit/labels/performances","name":"performances","color":"fbca04"}],"state":"closed","locked":false,"assignee":{"login":"ColinDuquesnoy","id":1681217,"avatar_url":"https://avatars.githubusercontent.com/u/1681217?v=3","gravatar_id":"","url":"https://api.github.com/users/ColinDuquesnoy","html_url":"https://github.com/ColinDuquesnoy","followers_url":"https://api.github.com/users/ColinDuquesnoy/followers","following_url":"https://api.github.com/users/ColinDuquesnoy/following{/other_user}","gists_url":"https://api.github.com/users/ColinDuquesnoy/gists{/gist_id}","starred_url":"https://api.github.com/users/ColinDuquesnoy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ColinDuquesnoy/subscriptions","organizations_url":"https://api.github.com/users/ColinDuquesnoy/orgs","repos_url":"https://api.github.com/users/ColinDuquesnoy/repos","events_url":"https://api.github.com/users/ColinDuquesnoy/events{/privacy}","received_events_url":"https://api.github.com/users/ColinDuquesnoy/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/HackEdit/hackedit/milestones/2","html_url":"https://github.com/HackEdit/hackedit/milestones/1.0a2","labels_url":"https://api.github.com/repos/HackEdit/hackedit/milestones/2/labels","id":1406458,"number":2,"title":"1.0a2","description":"Next future alpha release (bug fix release).","creator":{"login":"ColinDuquesnoy","id":1681217,"avatar_url":"https://avatars.githubusercontent.com/u/1681217?v=3","gravatar_id":"","url":"https://api.github.com/users/ColinDuquesnoy","html_url":"https://github.com/ColinDuquesnoy","followers_url":"https://api.github.com/users/ColinDuquesnoy/followers","following_url":"https://api.github.com/users/ColinDuquesnoy/following{/other_user}","gists_url":"https://api.github.com/users/ColinDuquesnoy/gists{/gist_id}","starred_url":"https://api.github.com/users/ColinDuquesnoy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ColinDuquesnoy/subscriptions","organizations_url":"https://api.github.com/users/ColinDuquesnoy/orgs","repos_url":"https://api.github.com/users/ColinDuquesnoy/repos","events_url":"https://api.github.com/users/ColinDuquesnoy/events{/privacy}","received_events_url":"https://api.github.com/users/ColinDuquesnoy/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":34,"state":"open","created_at":"2015-11-11T12:33:13Z","updated_at":"2016-03-25T08:49:47Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2015-11-13T17:31:35Z","updated_at":"2016-03-29T07:00:41Z","closed_at":"2016-03-23T08:31:12Z","body":"The current indexing system needs a serious overhaul. In short, the current implementation fall short for very big project (that contains GB of project data).\r\n\r\n### How indexing work at the moment:\r\n\r\nIndexing is triggered on the whole project when you start HE and open a project, when you add another project to the list of open projects or when you mark an item as ignored in the project view. A partial indexing is also triggered when you save a file (only on the file that changed\r\n\r\nThe project indexing is made in two steps:\r\n\r\n1. First we recursively walk trough the open project directories to build a list of all the relevant project files (following the ignore rules set in the mimetypes preferences or by ignoring them in the project tree view).\r\n2. Then we scan the list of project files. Each parse-able file is then parse to extract the list of definitions/symbols.\r\n\r\nIndexing is performed in a background process and the results are stored in a plain json file (``.hackedit/cache.usr``).\r\n\r\nWhen you use the goto actions, we load the entire project file list (or symbol tables depending on the goto action invoked). The list/table is then filtered and displayed in a QListWidget.\r\n\r\n### Issues\r\n\r\nThere multiple issues with this implementation:\r\n\r\n   - loading the entire project files/symbols takes a lot memory and cpu usage. This basically makes the GUI unresponsive and prone to crash on low spec systems.\r\n   - indexing is very dumb and very inefficient at the moment\r\n   - only project files are indexed. But with a more robust implementation we could expand indexing to scan other interesting sources such as the python library of the default project interpreter in ``hackedit-python`` or the copybook paths of the main project compiler in ``hackedit-cobol``,...\r\n\r\n### What needs to be changed\r\n\r\n1. [x] First we need a real database for storing all those data. We need something light, portable and easy to install/bundle. SQLite seems to be the most interesting solution because python has builtin support for it in the standard library (no additional dependency = easier install).\r\n2. [x]  We need to improve the indexing logic. For example by storing and comparing the time stamp of the files in order to avoid re-indexing things that have not changed.\r\n3. [x]  We should probably break indexing in two distinct phases: \r\n\r\n     - project indexing: only project files are indexed\r\n     - workspace specific indexing: indexing the other interesting source for the current workspace.\r\n\r\n    The result of the project indexation would end up in ``.hackedit/project.db`` while the results of   the workspace indexation would end up in ``~/.hackedit/$WORKSPACE_NAME.db``.\r\n4. [x]  We need to improve the GUI. We should avoid loading the entire database when there is no filter and wait for the user to validate the input before querying the database (or use a timer to delay the execution of the request until the user stopped writing). The request should probably be done in a background thread/process to keep a responsive UI.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/HackEdit/hackedit/issues/comments/202745921","html_url":"https://github.com/HackEdit/hackedit/issues/26#issuecomment-202745921","issue_url":"https://api.github.com/repos/HackEdit/hackedit/issues/26","id":202745921,"user":{"login":"ColinDuquesnoy","id":1681217,"avatar_url":"https://avatars.githubusercontent.com/u/1681217?v=3","gravatar_id":"","url":"https://api.github.com/users/ColinDuquesnoy","html_url":"https://github.com/ColinDuquesnoy","followers_url":"https://api.github.com/users/ColinDuquesnoy/followers","following_url":"https://api.github.com/users/ColinDuquesnoy/following{/other_user}","gists_url":"https://api.github.com/users/ColinDuquesnoy/gists{/gist_id}","starred_url":"https://api.github.com/users/ColinDuquesnoy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ColinDuquesnoy/subscriptions","organizations_url":"https://api.github.com/users/ColinDuquesnoy/orgs","repos_url":"https://api.github.com/users/ColinDuquesnoy/repos","events_url":"https://api.github.com/users/ColinDuquesnoy/events{/privacy}","received_events_url":"https://api.github.com/users/ColinDuquesnoy/received_events","type":"User","site_admin":false},"created_at":"2016-03-29T07:00:41Z","updated_at":"2016-03-29T07:00:41Z","body":"> Where is the sqlite database file(s) stored?\r\n\r\n- Windows: ``APPDATA/HackEdit``\r\n- GNU/Linux: ``~/.hackedit``\r\n- Mac OSX: ``~/Library/Application Support/HackEdit``"}}