{"action":"created","issue":{"url":"https://api.github.com/repos/apache/drill/issues/1250","repository_url":"https://api.github.com/repos/apache/drill","labels_url":"https://api.github.com/repos/apache/drill/issues/1250/labels{/name}","comments_url":"https://api.github.com/repos/apache/drill/issues/1250/comments","events_url":"https://api.github.com/repos/apache/drill/issues/1250/events","html_url":"https://github.com/apache/drill/pull/1250","id":320423095,"number":1250,"title":"DRILL-5270: Improve loading of profiles listing in the WebUI","user":{"login":"kkhatua","id":4335237,"avatar_url":"https://avatars0.githubusercontent.com/u/4335237?v=4","gravatar_id":"","url":"https://api.github.com/users/kkhatua","html_url":"https://github.com/kkhatua","followers_url":"https://api.github.com/users/kkhatua/followers","following_url":"https://api.github.com/users/kkhatua/following{/other_user}","gists_url":"https://api.github.com/users/kkhatua/gists{/gist_id}","starred_url":"https://api.github.com/users/kkhatua/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kkhatua/subscriptions","organizations_url":"https://api.github.com/users/kkhatua/orgs","repos_url":"https://api.github.com/users/kkhatua/repos","events_url":"https://api.github.com/users/kkhatua/events{/privacy}","received_events_url":"https://api.github.com/users/kkhatua/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-05-04T20:42:06Z","updated_at":"2018-05-08T03:44:56Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/drill/pulls/1250","html_url":"https://github.com/apache/drill/pull/1250","diff_url":"https://github.com/apache/drill/pull/1250.diff","patch_url":"https://github.com/apache/drill/pull/1250.patch"},"body":"_**Note:** Closed the old PR #755 and opening this._\r\n\r\nWhen Drill is displaying profiles stored on the file system (Local or Distributed), it does so by loading the entire list of `.sys.drill` files in the profile directory, sorting and deserializing. This can get expensive, since only a single CPU thread does this.\r\nAs an example, a directory of 120K profiles, the time to just fetch the list of files alone is over 6 seconds. After that, based on the number of profiles being rendered, the time varies. An average of 30ms is needed to deserialize a standard profile, which translates to an additional 3sec for therendering of default 100 profiles.\r\n\r\nA user reported issue confirms just that:\r\nDRILL-5028 Opening profiles page from web ui gets very slow when a lot of history files have been stored in HDFS or Local FS\r\n\r\nAdditional JIRAs filed ask for managing these profiles\r\nDRILL-2362 Drill should manage Query Profiling archiving\r\nDRILL-2861 enhance drill profile file management\r\n\r\nThis PR brings the following enhancements to achieve that:\r\n1. Mimick the In-memory persistence of profiles (DRILL-5481), by keeping only a predefined `max-capacity` number of profiles in the directory and moving the oldest to an 'archived' sub-directory.\r\n2. Improve loading times by pinning the deserialized list in memory (TreeSet; for maintaining a memory-efficient sortedness of the profiles). That way, if we do not detect any new profiles in the profileStore (i.e. profile directory) since the last time a web-request for rendering the profiles was made, we can re-serve the same listing and skip making a trip to the filesystem to re-fetch all the profiles.\r\n\r\nReload & reconstruction of the profiles in the Tree is done in the event of any of the following states changing:\r\n  i.   Modification Time of profile dir\r\n  ii.  Number of profiles in the profile dir\r\n  iii. Number of profiles requested exceeds existing the currently available list\r\n\r\n3. When 2 or more web-requests for rendering arrive, the WebServer code already processes the requests sequentially. As a result, the earliest request will trigger the reconstruction of the in-memory profile-set, and the last-modified timestamp of the profileStore is tracked. This way, the remaining blocked requests can re-use the freshly-reconstructed profile-set for rendering if the underlying profileStore has not been modified. There is an assumption made here that the rate of profiles being added to the profileStore is not too high to trigger a reconstruction for every queued up request. \r\n4. To prevent frequent archiving, there is a threshold (max-capacity) defined for triggering the archive. However, the number of profiles archived is selected to ensure that the profiles not archived is 90% of the threshold.\r\n5. To prevent the archiving process from taking too long, an archival rate (`drill.exec.profiles.store.archive.rate`) is defined so that upto that many number of profiles are archived in one go, before resumption of re-rendering takes place.\r\n6. On a Distributed FileSystem (e.g. HDFS), multiple Drillbits might attempt to archive. To mitigate that, if a Drillbit detects that it is unable to archive a profile, it will assume that another Drillbit is also archiving, and stop archiving any more."},"comment":{"url":"https://api.github.com/repos/apache/drill/issues/comments/387275507","html_url":"https://github.com/apache/drill/pull/1250#issuecomment-387275507","issue_url":"https://api.github.com/repos/apache/drill/issues/1250","id":387275507,"user":{"login":"ilooner","id":8020165,"avatar_url":"https://avatars1.githubusercontent.com/u/8020165?v=4","gravatar_id":"","url":"https://api.github.com/users/ilooner","html_url":"https://github.com/ilooner","followers_url":"https://api.github.com/users/ilooner/followers","following_url":"https://api.github.com/users/ilooner/following{/other_user}","gists_url":"https://api.github.com/users/ilooner/gists{/gist_id}","starred_url":"https://api.github.com/users/ilooner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ilooner/subscriptions","organizations_url":"https://api.github.com/users/ilooner/orgs","repos_url":"https://api.github.com/users/ilooner/repos","events_url":"https://api.github.com/users/ilooner/events{/privacy}","received_events_url":"https://api.github.com/users/ilooner/received_events","type":"User","site_admin":false},"created_at":"2018-05-08T03:44:56Z","updated_at":"2018-05-08T03:44:56Z","author_association":"CONTRIBUTOR","body":"@kkhatua Why not use the Guava Cache? http://www.baeldung.com/guava-cache . I think it would simplify the implementation."}}
{"actor":{"display_login":"ilooner"}}