{"action":"created","issue":{"url":"https://api.github.com/repos/LMFDB/lmfdb/issues/1814","repository_url":"https://api.github.com/repos/LMFDB/lmfdb","labels_url":"https://api.github.com/repos/LMFDB/lmfdb/issues/1814/labels{/name}","comments_url":"https://api.github.com/repos/LMFDB/lmfdb/issues/1814/comments","events_url":"https://api.github.com/repos/LMFDB/lmfdb/issues/1814/events","html_url":"https://github.com/LMFDB/lmfdb/pull/1814","id":165997564,"number":1814,"title":"Store statistics in the database","user":{"login":"AndrewVSutherland","id":11425002,"avatar_url":"https://avatars.githubusercontent.com/u/11425002?v=3","gravatar_id":"","url":"https://api.github.com/users/AndrewVSutherland","html_url":"https://github.com/AndrewVSutherland","followers_url":"https://api.github.com/users/AndrewVSutherland/followers","following_url":"https://api.github.com/users/AndrewVSutherland/following{/other_user}","gists_url":"https://api.github.com/users/AndrewVSutherland/gists{/gist_id}","starred_url":"https://api.github.com/users/AndrewVSutherland/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AndrewVSutherland/subscriptions","organizations_url":"https://api.github.com/users/AndrewVSutherland/orgs","repos_url":"https://api.github.com/users/AndrewVSutherland/repos","events_url":"https://api.github.com/users/AndrewVSutherland/events{/privacy}","received_events_url":"https://api.github.com/users/AndrewVSutherland/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-07-17T21:37:32Z","updated_at":"2016-07-18T09:20:13Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/LMFDB/lmfdb/pulls/1814","html_url":"https://github.com/LMFDB/lmfdb/pull/1814","diff_url":"https://github.com/LMFDB/lmfdb/pull/1814.diff","patch_url":"https://github.com/LMFDB/lmfdb/pull/1814.patch"},"body":"This is a first step toward addressing #1236.  It consists of two parts:\r\n\r\n- A new function update_attribute_stats has been added to lmfdb/data_mgt/utilities/rewrite.py. It takes a db, a collection name (say coll), and a list of attribute names.  For each attribute in the list, it creates a record in the collection coll.stats (which will be created if not present) that contains statistics for the attribute, including a sorted list of (value,count) pairs that counts how many objects in coll have attr set to value.  This is intended for attributes whose values lie in a reasonably small set (e.g. torsion structures, rank, etc...) on which one might want to report statistics (but even an attribute like \"base_field\" that might take hundreds of values should still be fine, it will work up to the pymongo bson limit of 16MB).  You can view an example at http://beta.lmfdb.org/api/genus2_curves/curves.stats/.\r\n\r\n- A new function attribute_value_counts has been added to lmfdb/utils.py that take a collection name and an attribute name.  It returns a sorted list of (value,count) pairs for the specified attribute as above.  If the collection coll.stats is present and contains a record for the specified attribute **and** the number of objects in coll is the same as it was when that record was created, it will return the list stored in coll.stats, otherwise it will compute it on the fly (if the statistics are present but out of date a warning will be written to flasklog suggesting that the statistics should be updated).\r\n\r\nYou can see this in action on http://127.0.0.1:37777/Genus2Curve/Q/stats, the genus 2 stats page has been updated to use attribute_value_counts when it computes statistics, which will read the stats from the database because they are present in genus2_curves.curves.stats.  You can compare to http://www.lmfdb.org/Genus2Curve/Q/stats to see that the statistics page looks the same.\r\n\r\nThis shaves about one second off the load time of for the genus 2 curves statistics page; the improvement should be greater for other statistics pages for larger collections if they are changed to use the new functions introduced in this PR.\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/LMFDB/lmfdb/issues/comments/233281896","html_url":"https://github.com/LMFDB/lmfdb/pull/1814#issuecomment-233281896","issue_url":"https://api.github.com/repos/LMFDB/lmfdb/issues/1814","id":233281896,"user":{"login":"AndrewVSutherland","id":11425002,"avatar_url":"https://avatars.githubusercontent.com/u/11425002?v=3","gravatar_id":"","url":"https://api.github.com/users/AndrewVSutherland","html_url":"https://github.com/AndrewVSutherland","followers_url":"https://api.github.com/users/AndrewVSutherland/followers","following_url":"https://api.github.com/users/AndrewVSutherland/following{/other_user}","gists_url":"https://api.github.com/users/AndrewVSutherland/gists{/gist_id}","starred_url":"https://api.github.com/users/AndrewVSutherland/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AndrewVSutherland/subscriptions","organizations_url":"https://api.github.com/users/AndrewVSutherland/orgs","repos_url":"https://api.github.com/users/AndrewVSutherland/repos","events_url":"https://api.github.com/users/AndrewVSutherland/events{/privacy}","received_events_url":"https://api.github.com/users/AndrewVSutherland/received_events","type":"User","site_admin":false},"created_at":"2016-07-18T09:20:13Z","updated_at":"2016-07-18T09:20:13Z","body":"@JohnCremona Good catch!  Fixed."}}
{"actor":{"display_login":"AndrewVSutherland"}}