{"action":"created","issue":{"url":"https://api.github.com/repos/18F/regulations-core/issues/35","repository_url":"https://api.github.com/repos/18F/regulations-core","labels_url":"https://api.github.com/repos/18F/regulations-core/issues/35/labels{/name}","comments_url":"https://api.github.com/repos/18F/regulations-core/issues/35/comments","events_url":"https://api.github.com/repos/18F/regulations-core/issues/35/events","html_url":"https://github.com/18F/regulations-core/pull/35","id":138361096,"number":35,"title":"[WIP] Prototype nested sets for regulations.","user":{"login":"jmcarp","id":1633460,"avatar_url":"https://avatars.githubusercontent.com/u/1633460?v=3","gravatar_id":"","url":"https://api.github.com/users/jmcarp","html_url":"https://github.com/jmcarp","followers_url":"https://api.github.com/users/jmcarp/followers","following_url":"https://api.github.com/users/jmcarp/following{/other_user}","gists_url":"https://api.github.com/users/jmcarp/gists{/gist_id}","starred_url":"https://api.github.com/users/jmcarp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmcarp/subscriptions","organizations_url":"https://api.github.com/users/jmcarp/orgs","repos_url":"https://api.github.com/users/jmcarp/repos","events_url":"https://api.github.com/users/jmcarp/events{/privacy}","received_events_url":"https://api.github.com/users/jmcarp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-03-04T02:55:12Z","updated_at":"2016-03-06T00:37:34Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/18F/regulations-core/pulls/35","html_url":"https://github.com/18F/regulations-core/pull/35","diff_url":"https://github.com/18F/regulations-core/pull/35.diff","patch_url":"https://github.com/18F/regulations-core/pull/35.patch"},"body":"This is incomplete, but I thought I'd put it out for feedback before finishing. Tasks:\r\n\r\n* [x] Use nested sets to store regulation hierarchy\r\n* [x] Avoid calling pseudo-private methods like `_transform`\r\n* [x] Restore `bulk_put` behavior if possible\r\n* [ ] Update unit tests\r\n\r\nMore specifically, it wound up being simpler to cast dicts to ORM records in the regulation write view, rather than transforming internally in `DMRegulation`. This would break the elastic data model without further changes, so that will have to get worked out before this is done.\r\n\r\nping @cmc333333 \r\n\r\n[Resolves #2]"},"comment":{"url":"https://api.github.com/repos/18F/regulations-core/issues/comments/192769160","html_url":"https://github.com/18F/regulations-core/pull/35#issuecomment-192769160","issue_url":"https://api.github.com/repos/18F/regulations-core/issues/35","id":192769160,"user":{"login":"cmc333333","id":326918,"avatar_url":"https://avatars.githubusercontent.com/u/326918?v=3","gravatar_id":"","url":"https://api.github.com/users/cmc333333","html_url":"https://github.com/cmc333333","followers_url":"https://api.github.com/users/cmc333333/followers","following_url":"https://api.github.com/users/cmc333333/following{/other_user}","gists_url":"https://api.github.com/users/cmc333333/gists{/gist_id}","starred_url":"https://api.github.com/users/cmc333333/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmc333333/subscriptions","organizations_url":"https://api.github.com/users/cmc333333/orgs","repos_url":"https://api.github.com/users/cmc333333/repos","events_url":"https://api.github.com/users/cmc333333/events{/privacy}","received_events_url":"https://api.github.com/users/cmc333333/received_events","type":"User","site_admin":false},"created_at":"2016-03-06T00:37:34Z","updated_at":"2016-03-06T00:37:34Z","body":"Pardon the rambling, but here goes anyway.\r\n\r\nSome context: the major impetus for this was finding 40 CFR 52, which is roughly as large as all of the other regs we've seen combined. We were already pushing the limit of what RDMS storage can happily accept (MySQL had a 1M message size at CFPB, iirc; we also ran the risk of the HTTP interface timing out). A reg that large surely would trigger both problems.\r\n\r\nRe S3/file storage: you are correct, the API has largely been a key-value store. It initially grew out of the need to search (hence the ElasticSearch storage backend) but was jammed into an RDMS due to technical policies. That said, I think there's a lot of relational data that we haven't made use of; it'd be super handy to know the most cited paragraph or section, for example, which is not easily answerable due to the key-value architecture. I'd prefer we move towards making the data easier to query than locking it up in binary blobs.\r\n\r\nOn static content in general, we've talked a lot about making the eRegs UI a static site generator. I think this would be a good direction to move as it's much easier to host and hand off to clients. We do have some dynamic components of the UI (particularly search), however, so we'd need to either completely disable these features or have some sort of mixed static-dynamic hybrid. In any event, that's a can of worms.\r\n\r\nUnfortunately, the API is pulled in two rather different directions. On one hand, we want to make it as flexible as possible, making a datastore that's easy to query; we want the API itself to be a product. On the other hand, there is only one consumer (the eRegs UI) and that has really molded the API's development to match its needs. This leads us to a rather annoying spot where we've implemented an API-first system, yet spent almost no time making a general purpose API. This has lead to duplication -- consider the parser's local \"index\", which should probably be backed by this API instead.\r\n\r\nAll of this is to say, I'm not sure the best approach. I think the storage savings are actually a big win in light of larger regulations, but that need isn't super pressing. I'd be content to close this PR and say \"now we know the strategy is viable\" and let sleeping dogs lie until we need to disturb them."}}