{"action":"created","issue":{"url":"https://api.github.com/repos/nilmtk/nilmtk/issues/13","repository_url":"https://api.github.com/repos/nilmtk/nilmtk","labels_url":"https://api.github.com/repos/nilmtk/nilmtk/issues/13/labels{/name}","comments_url":"https://api.github.com/repos/nilmtk/nilmtk/issues/13/comments","events_url":"https://api.github.com/repos/nilmtk/nilmtk/issues/13/events","html_url":"https://github.com/nilmtk/nilmtk/issues/13","id":23710337,"number":13,"title":"Design of our standard on-disk dataset format?","user":{"login":"JackKelly","id":460756,"avatar_url":"https://avatars2.githubusercontent.com/u/460756?v=4","gravatar_id":"","url":"https://api.github.com/users/JackKelly","html_url":"https://github.com/JackKelly","followers_url":"https://api.github.com/users/JackKelly/followers","following_url":"https://api.github.com/users/JackKelly/following{/other_user}","gists_url":"https://api.github.com/users/JackKelly/gists{/gist_id}","starred_url":"https://api.github.com/users/JackKelly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JackKelly/subscriptions","organizations_url":"https://api.github.com/users/JackKelly/orgs","repos_url":"https://api.github.com/users/JackKelly/repos","events_url":"https://api.github.com/users/JackKelly/events{/privacy}","received_events_url":"https://api.github.com/users/JackKelly/received_events","type":"User","site_admin":false},"labels":[{"id":70479570,"url":"https://api.github.com/repos/nilmtk/nilmtk/labels/Building","name":"Building","color":"d4c5f9","default":false},{"id":69006623,"url":"https://api.github.com/repos/nilmtk/nilmtk/labels/design","name":"design","color":"fbca04","default":false},{"id":68847511,"url":"https://api.github.com/repos/nilmtk/nilmtk/labels/question","name":"question","color":"cc317c","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/nilmtk/nilmtk/milestones/1","html_url":"https://github.com/nilmtk/nilmtk/milestone/1","labels_url":"https://api.github.com/repos/nilmtk/nilmtk/milestones/1/labels","id":503775,"number":1,"title":"first release","description":"All the features we want implemented and bugs that need to be fixed before the \"first full release\" ;)","creator":{"login":"JackKelly","id":460756,"avatar_url":"https://avatars2.githubusercontent.com/u/460756?v=4","gravatar_id":"","url":"https://api.github.com/users/JackKelly","html_url":"https://github.com/JackKelly","followers_url":"https://api.github.com/users/JackKelly/followers","following_url":"https://api.github.com/users/JackKelly/following{/other_user}","gists_url":"https://api.github.com/users/JackKelly/gists{/gist_id}","starred_url":"https://api.github.com/users/JackKelly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JackKelly/subscriptions","organizations_url":"https://api.github.com/users/JackKelly/orgs","repos_url":"https://api.github.com/users/JackKelly/repos","events_url":"https://api.github.com/users/JackKelly/events{/privacy}","received_events_url":"https://api.github.com/users/JackKelly/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":57,"state":"closed","created_at":"2013-12-03T15:50:07Z","updated_at":"2014-11-24T16:47:54Z","due_on":"2014-01-21T08:00:00Z","closed_at":"2014-10-20T09:34:55Z"},"comments":36,"created_at":"2013-12-04T11:49:09Z","updated_at":"2018-03-21T13:28:33Z","closed_at":"2014-07-12T02:45:58Z","author_association":"CONTRIBUTOR","body":"Just some basic questions about our on-disk dataset format.\n#### Name?\n\nName?  REDD+ is nicely descriptive (because our format will be an extension of REDD), but I worry that this name might, um, have political implications?  We should perhaps approach the REDD authors before committing to this name (if indeed we like the name!)\n#### Standardise units and names\n\nShould our format standardise the following:\n- standardise units (use unix timestamp (UTC) for timestamps, Watts, Volts, etc)\n- standardise column names for aggregate data (e.g. 'main0_apparent', 'mains1_active')\n- standardise appliance names (perhaps based on the HES appliance naming scheme with apparent/active/reactive appended to the name like kettle_active?)\n\nSo, for example, once data is in our format, users will know for sure that the kettle appliance is called 'kettle_active' and not 'Kettle' or 'KETTLE0' or something.\n\nIt would be the responsibility of functions like `HES.load_buiding` to convert from HES's units to our standard units.\n\nI think we should standardise.  It feels like The Right Thing To Do; and it should minimise the complexity of our NILM code.\n#### Storing power data on disk\n\nWe can probably just use the basic REDD structure (a directory per house; a labels.dat file for channel labels etc).\n\nI think we should allow users the option of saving either CSV or HSF5.  CSV files can be slow to load for large datasets (even from an SSD).\n\nIt would be good if our format could be loaded by existing tools written to process REDD.  i.e. anything we add (e.g. metadata) should be in separate files.\n#### Storing metadata on disk\n\nIt sounds like we're thinking of using JSON.  Is that OK?  Maybe call the file 'metadata.json'.  I guess we need a metadata file for the dataset as a whole (storing country of origin, academic references, URL, nominal mains voltage etc) and a metadata file per house (geo location, number of occupants etc).  How does that sound?\n\nThe metadata ideas described by @nipunreddevil in the wiki look very sensible.  I wonder if we can also use some ideas from [Xively's metadata spec](https://xively.com/dev/docs/api/quick_reference/api_resource_attributes/) and/or [SensorML](http://en.wikipedia.org/wiki/SensorML).  And this blog has some interesting and up-to-date ideas: [Data models for the Internet of Things](http://iot-datamodels.blogspot.co.uk/)\n#### How to cope with BLUED?\n\nAs @oliparson pointed out yesterday, BLUED  doesn't record the power consumption of individual appliances.  Instead it records whether appliances are on or off.  Trying to estimate the individual appliance power consumption is bound to be prone to large uncertainties so I think we shouldn't do that.  Instead, it may be simpler to add a new class of file: `channel_X_state.dat` that has two columns: a timestamp and a state (an integer).  We could also use this format for the exporting appliance estimates from NILM algorithms which know about separate appliance states.\n#### Allow multiple columns per file?\n\nFor mains, my dataset records active, apparent and voltage for aggregate. It's much for efficient to store this in a single csv file with 4 columns rather than 3 csv files (because the timestamp must be duplicated in each file) . Should we allow multiple columns in our on disk  format? \n#### Compact representation?\n\nThe standard REDD format is quite space inefficient, especially if we use it for storing appliance state.  e.g.:\n\n```\n1303132929 0\n1303132930 0\n1303132931 0\n...\n1303170000 0\n1303170001 1\n...\n```\n\nSo perhaps we could allow users to export CSV files as either 'complete' or 'compact' (where only change points would be reported).\n"},"comment":{"url":"https://api.github.com/repos/nilmtk/nilmtk/issues/comments/374935963","html_url":"https://github.com/nilmtk/nilmtk/issues/13#issuecomment-374935963","issue_url":"https://api.github.com/repos/nilmtk/nilmtk/issues/13","id":374935963,"user":{"login":"nipunbatra","id":60985,"avatar_url":"https://avatars2.githubusercontent.com/u/60985?v=4","gravatar_id":"","url":"https://api.github.com/users/nipunbatra","html_url":"https://github.com/nipunbatra","followers_url":"https://api.github.com/users/nipunbatra/followers","following_url":"https://api.github.com/users/nipunbatra/following{/other_user}","gists_url":"https://api.github.com/users/nipunbatra/gists{/gist_id}","starred_url":"https://api.github.com/users/nipunbatra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nipunbatra/subscriptions","organizations_url":"https://api.github.com/users/nipunbatra/orgs","repos_url":"https://api.github.com/users/nipunbatra/repos","events_url":"https://api.github.com/users/nipunbatra/events{/privacy}","received_events_url":"https://api.github.com/users/nipunbatra/received_events","type":"User","site_admin":false},"created_at":"2018-03-21T13:28:33Z","updated_at":"2018-03-21T13:28:33Z","author_association":"OWNER","body":"@Zouhour03 NILMTK doesn't support Blued like datasets. Please contact the dataset authors in case you have other clarifications."}}
{"actor":{"display_login":"nipunbatra"}}