{"action":"created","issue":{"url":"https://api.github.com/repos/h2oai/datatable/issues/1109","repository_url":"https://api.github.com/repos/h2oai/datatable","labels_url":"https://api.github.com/repos/h2oai/datatable/issues/1109/labels{/name}","comments_url":"https://api.github.com/repos/h2oai/datatable/issues/1109/comments","events_url":"https://api.github.com/repos/h2oai/datatable/issues/1109/events","html_url":"https://github.com/h2oai/datatable/issues/1109","id":331404952,"node_id":"MDU6SXNzdWUzMzE0MDQ5NTI=","number":1109,"title":"New binary format: Jay","user":{"login":"st-pasha","id":4231472,"node_id":"MDQ6VXNlcjQyMzE0NzI=","avatar_url":"https://avatars0.githubusercontent.com/u/4231472?v=4","gravatar_id":"","url":"https://api.github.com/users/st-pasha","html_url":"https://github.com/st-pasha","followers_url":"https://api.github.com/users/st-pasha/followers","following_url":"https://api.github.com/users/st-pasha/following{/other_user}","gists_url":"https://api.github.com/users/st-pasha/gists{/gist_id}","starred_url":"https://api.github.com/users/st-pasha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/st-pasha/subscriptions","organizations_url":"https://api.github.com/users/st-pasha/orgs","repos_url":"https://api.github.com/users/st-pasha/repos","events_url":"https://api.github.com/users/st-pasha/events{/privacy}","received_events_url":"https://api.github.com/users/st-pasha/received_events","type":"User","site_admin":false},"labels":[{"id":795224487,"node_id":"MDU6TGFiZWw3OTUyMjQ0ODc=","url":"https://api.github.com/repos/h2oai/datatable/labels/design-doc","name":"design-doc","color":"fbfbfb","default":false}],"state":"open","locked":false,"assignee":{"login":"st-pasha","id":4231472,"node_id":"MDQ6VXNlcjQyMzE0NzI=","avatar_url":"https://avatars0.githubusercontent.com/u/4231472?v=4","gravatar_id":"","url":"https://api.github.com/users/st-pasha","html_url":"https://github.com/st-pasha","followers_url":"https://api.github.com/users/st-pasha/followers","following_url":"https://api.github.com/users/st-pasha/following{/other_user}","gists_url":"https://api.github.com/users/st-pasha/gists{/gist_id}","starred_url":"https://api.github.com/users/st-pasha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/st-pasha/subscriptions","organizations_url":"https://api.github.com/users/st-pasha/orgs","repos_url":"https://api.github.com/users/st-pasha/repos","events_url":"https://api.github.com/users/st-pasha/events{/privacy}","received_events_url":"https://api.github.com/users/st-pasha/received_events","type":"User","site_admin":false},"assignees":[{"login":"st-pasha","id":4231472,"node_id":"MDQ6VXNlcjQyMzE0NzI=","avatar_url":"https://avatars0.githubusercontent.com/u/4231472?v=4","gravatar_id":"","url":"https://api.github.com/users/st-pasha","html_url":"https://github.com/st-pasha","followers_url":"https://api.github.com/users/st-pasha/followers","following_url":"https://api.github.com/users/st-pasha/following{/other_user}","gists_url":"https://api.github.com/users/st-pasha/gists{/gist_id}","starred_url":"https://api.github.com/users/st-pasha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/st-pasha/subscriptions","organizations_url":"https://api.github.com/users/st-pasha/orgs","repos_url":"https://api.github.com/users/st-pasha/repos","events_url":"https://api.github.com/users/st-pasha/events{/privacy}","received_events_url":"https://api.github.com/users/st-pasha/received_events","type":"User","site_admin":false}],"milestone":null,"comments":4,"created_at":"2018-06-12T01:27:54Z","updated_at":"2018-06-14T07:07:57Z","closed_at":null,"author_association":"COLLABORATOR","body":"This proposal suggests switching from the current \"NFF\" binary format for saving datatable Frames on disk, to a new format codenamed `Jay`. The new format saves the data into a single file (not folder) and keeps meta information in `Cap'n Proto` format (not CSV).\r\n\r\nIt is believed that the new format will be an improvement over NFF in several respects:\r\n1. Single file is easier to pass around from one machine to another.\r\n2. For a dataset containing multiple columns (10s or 100s of thousands), the operating system becomes very sluggish with any file-related operation.\r\n3. Frames containing many columns but few rows waste a lot of disk space in NFF format due to each file having a minimum size of 4096 bytes.\r\n4. A frame containing `n` columns requires between `n` and `2n` open file handles or memmapped regions, which becomes difficult on systems where those resources are limited. This becomes especially problematic in row-by-row operations on the frame.\r\n5. The architecture that forces us to keep many memmap regions opened at the same time is prone to sporadic memory errors.\r\n6. Cap'n Proto storage of meta information is more flexible and can be parsed faster than CSV.\r\n\r\nThe NFF format has the following advantages over Jay:\r\n1. Multi-file storage is advantageous over single-file for gigantic datasets (100s of GB)\r\n2. One file per column allows an administrator to control permissions for each column.\r\n3. Having one file per column allows extending the dataset with new data more easily.\r\n\r\nIn short, NFF has several purely theoretical advantages, whereas Jay has many practical ones.\r\n\r\n## Jay format\r\n```\r\n  \"Jay1\\0\\0\\0\\0\"       -- 8 bytes\r\n  [data section]       -- ...\r\n  [meta section]       -- <meta size>\r\n  <meta size>          -- 8 bytes\r\n  \"\\0\\0\\0\\0Jay1\"       -- 8 bytes\r\n```\r\n* The size of the file must be a multiple of 8 bytes, and all sections are aligned at 8-byte boundaries.\r\n* The file begins and ends with `\"Jay1\"` string padded with 4 NUL bytes (`\\0`). \r\n* Eight bytes before the last contain the size of the meta section (as an `int64` written in little-endian format). \r\n* The \"meta section\" is at the offset `16 + meta_size` from the end of the file, and it contains all meta information about the frame (including all column descriptors) serialized in Cap'n Proto format. \r\n* The portion of the file starting at offset 8 and ending at the beginning of the meta section is considered the data section. This section contains data buffers for each column. The exact location of each data buffer within the data section is stored in the meta section. The data buffers need not be adjacent to each other and are not necessarily stored in the same order as the columns themselves. However, each data buffer must be aligned at the 8-byte boundary.\r\n\r\n## Saving a frame into a Jay file\r\n* Create a \"meta info\" Cap'n Proto object\r\n* For each column in the frame, write its buffers into the file, and its meta information into the meta object (including the locations of each buffer).\r\n* Once all columns are written, write the meta object into the file\r\n* Write the size of the meta, and the final signature.\r\n\r\n## Opening a Jay file\r\n* Open the file and verify its \"Jay1\" signatures at the beginning/end\r\n* Read the meta size at the end of the file.\r\n* Read the meta object with the information about the frame and all columns.\r\n* At this point there are 2 choices: we can either memmap the entire file, and create each Column object as a subview into the large memmapped region, or memmap each portion of the file into a separate lazy column (similar how we do with NFF).\r\n\r\n## Additional remarks\r\n* This proposal does not yet specify the exact structure of the meta information, but generally it will follow the internal representation of the `DataTable` object.\r\n* #1054 is a pre-requisite.\r\n* #958 seeks to change the internal representation of string columns, and is better to do together with the switch to Jay, so as to avoid supporting too many different formats at the same time.\r\n* New functionality will be exposed via standard `dt.open()` / `Frame.save()`. Ability to open NFF format will still be available for some time with `dt.open_nff()`.\r\n* Issue #112 will become irrelevant."},"comment":{"url":"https://api.github.com/repos/h2oai/datatable/issues/comments/397194166","html_url":"https://github.com/h2oai/datatable/issues/1109#issuecomment-397194166","issue_url":"https://api.github.com/repos/h2oai/datatable/issues/1109","id":397194166,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NzE5NDE2Ng==","user":{"login":"jangorecki","id":3627377,"node_id":"MDQ6VXNlcjM2MjczNzc=","avatar_url":"https://avatars3.githubusercontent.com/u/3627377?v=4","gravatar_id":"","url":"https://api.github.com/users/jangorecki","html_url":"https://github.com/jangorecki","followers_url":"https://api.github.com/users/jangorecki/followers","following_url":"https://api.github.com/users/jangorecki/following{/other_user}","gists_url":"https://api.github.com/users/jangorecki/gists{/gist_id}","starred_url":"https://api.github.com/users/jangorecki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jangorecki/subscriptions","organizations_url":"https://api.github.com/users/jangorecki/orgs","repos_url":"https://api.github.com/users/jangorecki/repos","events_url":"https://api.github.com/users/jangorecki/events{/privacy}","received_events_url":"https://api.github.com/users/jangorecki/received_events","type":"User","site_admin":false},"created_at":"2018-06-14T07:07:57Z","updated_at":"2018-06-14T07:07:57Z","author_association":"MEMBER","body":"> One advantage of NFF (one file per column) is that columns can be added and deleted very easily and robustly.\r\n\r\nThis kind of model is even used in databases and is called 6NF (6th normal form). Ability to add/delete columns without altering table object is called _non destructive schema evolution_ and is crucial feature of Anchor Modeling (6NF design technique). A quote from wikipedia:\r\n> Anchor modeling is an agile database modeling technique suited for information that changes over time both in structure and content.\r\n\r\nWhen switching to single file model we are losing this ability.\r\n@stephlocke is expert on that :)\r\n\r\n---\r\n\r\nI also agree that thousands of columns are not that common cases in SQL RDBMS, from my experience in big warehouses I was having 200-1000 columns. When having more then better to melt your data and transform columns into rows to process by group.\r\nI can imaging there are use cases in ML where you need to have as _flat_ data as possible, like _one-hot encoding_. This justify to implement efficient backend for big amounts of columns, but at the same time I don't think it make sense to drop existing __\"no format\"__ model.\r\n\r\n---\r\n\r\nBefore making decisions it will be best to see benchmark of both."}}
{"actor":{"display_login":"jangorecki"}}