{"action":"closed","number":5065,"pull_request":{"url":"https://api.github.com/repos/astropy/astropy/pulls/5065","id":73377762,"html_url":"https://github.com/astropy/astropy/pull/5065","diff_url":"https://github.com/astropy/astropy/pull/5065.diff","patch_url":"https://github.com/astropy/astropy/pull/5065.patch","issue_url":"https://api.github.com/repos/astropy/astropy/issues/5065","number":5065,"state":"closed","locked":false,"title":"Lazy loading of HDUs on fits.open","user":{"login":"embray","id":676149,"avatar_url":"https://avatars.githubusercontent.com/u/676149?v=3","gravatar_id":"","url":"https://api.github.com/users/embray","html_url":"https://github.com/embray","followers_url":"https://api.github.com/users/embray/followers","following_url":"https://api.github.com/users/embray/following{/other_user}","gists_url":"https://api.github.com/users/embray/gists{/gist_id}","starred_url":"https://api.github.com/users/embray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/embray/subscriptions","organizations_url":"https://api.github.com/users/embray/orgs","repos_url":"https://api.github.com/users/embray/repos","events_url":"https://api.github.com/users/embray/events{/privacy}","received_events_url":"https://api.github.com/users/embray/received_events","type":"User","site_admin":false},"body":"This is something I've wanted to do a long time.  And while this isn't exactly how I had in mind to do it, it lays a lot of groundwork.  This is my alternative to #4634 \n\nThe idea here is that `fits.open()` currently immediately seeks through an entire FITS file, and finds all HDUs, as well as creates HDU objects and more importantly Header objects for them.  In a typical FITS file this overhead is negligible.  But for files containing hundreds of HDUs this can have a serious impact.  Further this can have a serious impact even when working with files of just a few HDUs, but when there are thousands of them that need to be looped over (and in particular when only the PRIMARY header is needed which is frequently the case).\n\nThis is an experimental attempt to add lazy-loading of HDUs, so HDUs are only read on demand.  For example if you open a file with `hdul = fits.open(filename)` and immediately do something like `hdul[3]` it read all HDUs up through the fourth one and then stop any further reading until necessary.\n\nOf course some operations, most notably `len(hdul)` and `hdul.info()` necessitate reading the entire file.  But these are not the most common cases in scripts which is what this change really concerns itself with (as opposed to command-line interaction).\n\nThe biggest shortcoming to this patch as it stands is the issue I mentioned in #4634 regarding `hdulist.close()`.  Because `fits.open()` used to load all HDUs, and in particular their headers, I have no doubt that there is code in the wild that follows this sort of pattern:\n\n```\n>>> hdulist = fits.open(filename)\n>>> # do some stuff\n>>> hdulist.close()\n>>> print(hdulist[3].header)\n```\n\nBecause of the lazy-loading, `hdulist[3]` may not have been loaded yet before `hdulist.close()` was called, and so this will fail.  I'm of the opinion that it _should_ fail--you shouldn't `close()` the file object until you're actually done doing everything you're going to do with it.  But unfortunately PyFITS has played loose on this forever.\n\nSo currently, to maintain backwards compatibility, this patch ensures that all HDUs are loaded before the file is closed.  This is of course kills most benefit from lazy-loading.  But not in all cases!  In special cases like convenience functions (`getheader`, `getdata`, etc.) in which the user never has direct access to the `HDUList` object we can still take full advantage of lazy-loading.  Furthermore, attempts to access a previously not loaded HDU _after_ the file has been closed will result in a `DeprecationWarning`, with the aim of disallowing this usage in the long term, so that lazy-loading can be fully taken advantage of.\n","created_at":"2016-06-10T13:58:23Z","updated_at":"2016-12-09T23:17:21Z","closed_at":"2016-12-09T23:17:21Z","merged_at":"2016-12-09T23:17:21Z","merge_commit_sha":"6ab68e2c560d7c45084f24b0310c25e677466eac","assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/astropy/astropy/milestones/38","html_url":"https://github.com/astropy/astropy/milestone/38","labels_url":"https://api.github.com/repos/astropy/astropy/milestones/38/labels","id":1761583,"number":38,"title":"v1.3.0","description":"","creator":{"login":"eteq","id":346587,"avatar_url":"https://avatars.githubusercontent.com/u/346587?v=3","gravatar_id":"","url":"https://api.github.com/users/eteq","html_url":"https://github.com/eteq","followers_url":"https://api.github.com/users/eteq/followers","following_url":"https://api.github.com/users/eteq/following{/other_user}","gists_url":"https://api.github.com/users/eteq/gists{/gist_id}","starred_url":"https://api.github.com/users/eteq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eteq/subscriptions","organizations_url":"https://api.github.com/users/eteq/orgs","repos_url":"https://api.github.com/users/eteq/repos","events_url":"https://api.github.com/users/eteq/events{/privacy}","received_events_url":"https://api.github.com/users/eteq/received_events","type":"User","site_admin":false},"open_issues":7,"closed_issues":165,"state":"open","created_at":"2016-05-12T04:39:02Z","updated_at":"2016-12-09T23:17:21Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/astropy/astropy/pulls/5065/commits","review_comments_url":"https://api.github.com/repos/astropy/astropy/pulls/5065/comments","review_comment_url":"https://api.github.com/repos/astropy/astropy/pulls/comments{/number}","comments_url":"https://api.github.com/repos/astropy/astropy/issues/5065/comments","statuses_url":"https://api.github.com/repos/astropy/astropy/statuses/6ab68e2c560d7c45084f24b0310c25e677466eac","head":{"label":"embray:fits/lazy-load-hdus","ref":"fits/lazy-load-hdus","sha":"6ab68e2c560d7c45084f24b0310c25e677466eac","user":{"login":"embray","id":676149,"avatar_url":"https://avatars.githubusercontent.com/u/676149?v=3","gravatar_id":"","url":"https://api.github.com/users/embray","html_url":"https://github.com/embray","followers_url":"https://api.github.com/users/embray/followers","following_url":"https://api.github.com/users/embray/following{/other_user}","gists_url":"https://api.github.com/users/embray/gists{/gist_id}","starred_url":"https://api.github.com/users/embray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/embray/subscriptions","organizations_url":"https://api.github.com/users/embray/orgs","repos_url":"https://api.github.com/users/embray/repos","events_url":"https://api.github.com/users/embray/events{/privacy}","received_events_url":"https://api.github.com/users/embray/received_events","type":"User","site_admin":false},"repo":{"id":2223452,"name":"astropy","full_name":"embray/astropy","owner":{"login":"embray","id":676149,"avatar_url":"https://avatars.githubusercontent.com/u/676149?v=3","gravatar_id":"","url":"https://api.github.com/users/embray","html_url":"https://github.com/embray","followers_url":"https://api.github.com/users/embray/followers","following_url":"https://api.github.com/users/embray/following{/other_user}","gists_url":"https://api.github.com/users/embray/gists{/gist_id}","starred_url":"https://api.github.com/users/embray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/embray/subscriptions","organizations_url":"https://api.github.com/users/embray/orgs","repos_url":"https://api.github.com/users/embray/repos","events_url":"https://api.github.com/users/embray/events{/privacy}","received_events_url":"https://api.github.com/users/embray/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/embray/astropy","description":"My fork of the main Astropy repository (astropy/astropy)","fork":true,"url":"https://api.github.com/repos/embray/astropy","forks_url":"https://api.github.com/repos/embray/astropy/forks","keys_url":"https://api.github.com/repos/embray/astropy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/embray/astropy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/embray/astropy/teams","hooks_url":"https://api.github.com/repos/embray/astropy/hooks","issue_events_url":"https://api.github.com/repos/embray/astropy/issues/events{/number}","events_url":"https://api.github.com/repos/embray/astropy/events","assignees_url":"https://api.github.com/repos/embray/astropy/assignees{/user}","branches_url":"https://api.github.com/repos/embray/astropy/branches{/branch}","tags_url":"https://api.github.com/repos/embray/astropy/tags","blobs_url":"https://api.github.com/repos/embray/astropy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/embray/astropy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/embray/astropy/git/refs{/sha}","trees_url":"https://api.github.com/repos/embray/astropy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/embray/astropy/statuses/{sha}","languages_url":"https://api.github.com/repos/embray/astropy/languages","stargazers_url":"https://api.github.com/repos/embray/astropy/stargazers","contributors_url":"https://api.github.com/repos/embray/astropy/contributors","subscribers_url":"https://api.github.com/repos/embray/astropy/subscribers","subscription_url":"https://api.github.com/repos/embray/astropy/subscription","commits_url":"https://api.github.com/repos/embray/astropy/commits{/sha}","git_commits_url":"https://api.github.com/repos/embray/astropy/git/commits{/sha}","comments_url":"https://api.github.com/repos/embray/astropy/comments{/number}","issue_comment_url":"https://api.github.com/repos/embray/astropy/issues/comments{/number}","contents_url":"https://api.github.com/repos/embray/astropy/contents/{+path}","compare_url":"https://api.github.com/repos/embray/astropy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/embray/astropy/merges","archive_url":"https://api.github.com/repos/embray/astropy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/embray/astropy/downloads","issues_url":"https://api.github.com/repos/embray/astropy/issues{/number}","pulls_url":"https://api.github.com/repos/embray/astropy/pulls{/number}","milestones_url":"https://api.github.com/repos/embray/astropy/milestones{/number}","notifications_url":"https://api.github.com/repos/embray/astropy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/embray/astropy/labels{/name}","releases_url":"https://api.github.com/repos/embray/astropy/releases{/id}","deployments_url":"https://api.github.com/repos/embray/astropy/deployments","created_at":"2011-08-17T18:24:15Z","updated_at":"2015-03-28T23:20:13Z","pushed_at":"2016-12-09T15:21:20Z","git_url":"git://github.com/embray/astropy.git","ssh_url":"git@github.com:embray/astropy.git","clone_url":"https://github.com/embray/astropy.git","svn_url":"https://github.com/embray/astropy","homepage":"astropy.org","size":58620,"stargazers_count":1,"watchers_count":1,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":1,"default_branch":"master"}},"base":{"label":"astropy:master","ref":"master","sha":"2cb4567eff9d09540e76024a54653c593d4a41c8","user":{"login":"astropy","id":847984,"avatar_url":"https://avatars.githubusercontent.com/u/847984?v=3","gravatar_id":"","url":"https://api.github.com/users/astropy","html_url":"https://github.com/astropy","followers_url":"https://api.github.com/users/astropy/followers","following_url":"https://api.github.com/users/astropy/following{/other_user}","gists_url":"https://api.github.com/users/astropy/gists{/gist_id}","starred_url":"https://api.github.com/users/astropy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/astropy/subscriptions","organizations_url":"https://api.github.com/users/astropy/orgs","repos_url":"https://api.github.com/users/astropy/repos","events_url":"https://api.github.com/users/astropy/events{/privacy}","received_events_url":"https://api.github.com/users/astropy/received_events","type":"Organization","site_admin":false},"repo":{"id":2081289,"name":"astropy","full_name":"astropy/astropy","owner":{"login":"astropy","id":847984,"avatar_url":"https://avatars.githubusercontent.com/u/847984?v=3","gravatar_id":"","url":"https://api.github.com/users/astropy","html_url":"https://github.com/astropy","followers_url":"https://api.github.com/users/astropy/followers","following_url":"https://api.github.com/users/astropy/following{/other_user}","gists_url":"https://api.github.com/users/astropy/gists{/gist_id}","starred_url":"https://api.github.com/users/astropy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/astropy/subscriptions","organizations_url":"https://api.github.com/users/astropy/orgs","repos_url":"https://api.github.com/users/astropy/repos","events_url":"https://api.github.com/users/astropy/events{/privacy}","received_events_url":"https://api.github.com/users/astropy/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/astropy/astropy","description":"Repository for the Astropy core package","fork":false,"url":"https://api.github.com/repos/astropy/astropy","forks_url":"https://api.github.com/repos/astropy/astropy/forks","keys_url":"https://api.github.com/repos/astropy/astropy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/astropy/astropy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/astropy/astropy/teams","hooks_url":"https://api.github.com/repos/astropy/astropy/hooks","issue_events_url":"https://api.github.com/repos/astropy/astropy/issues/events{/number}","events_url":"https://api.github.com/repos/astropy/astropy/events","assignees_url":"https://api.github.com/repos/astropy/astropy/assignees{/user}","branches_url":"https://api.github.com/repos/astropy/astropy/branches{/branch}","tags_url":"https://api.github.com/repos/astropy/astropy/tags","blobs_url":"https://api.github.com/repos/astropy/astropy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/astropy/astropy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/astropy/astropy/git/refs{/sha}","trees_url":"https://api.github.com/repos/astropy/astropy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/astropy/astropy/statuses/{sha}","languages_url":"https://api.github.com/repos/astropy/astropy/languages","stargazers_url":"https://api.github.com/repos/astropy/astropy/stargazers","contributors_url":"https://api.github.com/repos/astropy/astropy/contributors","subscribers_url":"https://api.github.com/repos/astropy/astropy/subscribers","subscription_url":"https://api.github.com/repos/astropy/astropy/subscription","commits_url":"https://api.github.com/repos/astropy/astropy/commits{/sha}","git_commits_url":"https://api.github.com/repos/astropy/astropy/git/commits{/sha}","comments_url":"https://api.github.com/repos/astropy/astropy/comments{/number}","issue_comment_url":"https://api.github.com/repos/astropy/astropy/issues/comments{/number}","contents_url":"https://api.github.com/repos/astropy/astropy/contents/{+path}","compare_url":"https://api.github.com/repos/astropy/astropy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/astropy/astropy/merges","archive_url":"https://api.github.com/repos/astropy/astropy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/astropy/astropy/downloads","issues_url":"https://api.github.com/repos/astropy/astropy/issues{/number}","pulls_url":"https://api.github.com/repos/astropy/astropy/pulls{/number}","milestones_url":"https://api.github.com/repos/astropy/astropy/milestones{/number}","notifications_url":"https://api.github.com/repos/astropy/astropy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/astropy/astropy/labels{/name}","releases_url":"https://api.github.com/repos/astropy/astropy/releases{/id}","deployments_url":"https://api.github.com/repos/astropy/astropy/deployments","created_at":"2011-07-21T01:33:49Z","updated_at":"2016-12-08T22:23:18Z","pushed_at":"2016-12-09T23:17:13Z","git_url":"git://github.com/astropy/astropy.git","ssh_url":"git@github.com:astropy/astropy.git","clone_url":"https://github.com/astropy/astropy.git","svn_url":"https://github.com/astropy/astropy","homepage":"www.astropy.org","size":70956,"stargazers_count":1072,"watchers_count":1072,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":619,"mirror_url":null,"open_issues_count":758,"forks":619,"open_issues":758,"watchers":1072,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/astropy/astropy/pulls/5065"},"html":{"href":"https://github.com/astropy/astropy/pull/5065"},"issue":{"href":"https://api.github.com/repos/astropy/astropy/issues/5065"},"comments":{"href":"https://api.github.com/repos/astropy/astropy/issues/5065/comments"},"review_comments":{"href":"https://api.github.com/repos/astropy/astropy/pulls/5065/comments"},"review_comment":{"href":"https://api.github.com/repos/astropy/astropy/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/astropy/astropy/pulls/5065/commits"},"statuses":{"href":"https://api.github.com/repos/astropy/astropy/statuses/6ab68e2c560d7c45084f24b0310c25e677466eac"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"eteq","id":346587,"avatar_url":"https://avatars.githubusercontent.com/u/346587?v=3","gravatar_id":"","url":"https://api.github.com/users/eteq","html_url":"https://github.com/eteq","followers_url":"https://api.github.com/users/eteq/followers","following_url":"https://api.github.com/users/eteq/following{/other_user}","gists_url":"https://api.github.com/users/eteq/gists{/gist_id}","starred_url":"https://api.github.com/users/eteq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eteq/subscriptions","organizations_url":"https://api.github.com/users/eteq/orgs","repos_url":"https://api.github.com/users/eteq/repos","events_url":"https://api.github.com/users/eteq/events{/privacy}","received_events_url":"https://api.github.com/users/eteq/received_events","type":"User","site_admin":false},"comments":27,"review_comments":10,"commits":11,"additions":411,"deletions":122,"changed_files":4}}
{"actor":{"display_login":"eteq"}}