{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/ml-cpp/issues/315","repository_url":"https://api.github.com/repos/elastic/ml-cpp","labels_url":"https://api.github.com/repos/elastic/ml-cpp/issues/315/labels{/name}","comments_url":"https://api.github.com/repos/elastic/ml-cpp/issues/315/comments","events_url":"https://api.github.com/repos/elastic/ml-cpp/issues/315/events","html_url":"https://github.com/elastic/ml-cpp/issues/315","id":380332032,"node_id":"MDU6SXNzdWUzODAzMzIwMzI=","number":315,"title":"[ML] Decisions relating to new analytics process input","user":{"login":"droberts195","id":7405510,"node_id":"MDQ6VXNlcjc0MDU1MTA=","avatar_url":"https://avatars0.githubusercontent.com/u/7405510?v=4","gravatar_id":"","url":"https://api.github.com/users/droberts195","html_url":"https://github.com/droberts195","followers_url":"https://api.github.com/users/droberts195/followers","following_url":"https://api.github.com/users/droberts195/following{/other_user}","gists_url":"https://api.github.com/users/droberts195/gists{/gist_id}","starred_url":"https://api.github.com/users/droberts195/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/droberts195/subscriptions","organizations_url":"https://api.github.com/users/droberts195/orgs","repos_url":"https://api.github.com/users/droberts195/repos","events_url":"https://api.github.com/users/droberts195/events{/privacy}","received_events_url":"https://api.github.com/users/droberts195/received_events","type":"User","site_admin":false},"labels":[{"id":865918316,"node_id":"MDU6TGFiZWw4NjU5MTgzMTY=","url":"https://api.github.com/repos/elastic/ml-cpp/labels/:ml","name":":ml","color":"009800","default":false},{"id":844906107,"node_id":"MDU6TGFiZWw4NDQ5MDYxMDc=","url":"https://api.github.com/repos/elastic/ml-cpp/labels/discuss","name":"discuss","color":"fbca04","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-13T17:11:41Z","updated_at":"2018-11-21T13:16:30Z","closed_at":"2018-11-21T13:16:30Z","author_association":"CONTRIBUTOR","body":"We have a [stub](https://github.com/elastic/ml-cpp/pull/298/files#diff-740e94a98f6c2a36c617970df2bc2c81R14) for handling data in the new `data_frame_analyzer` analytics process.  Currently this is receiving input in the form of an unordered map of named fields.\r\n\r\nWe've said we want to have the option to use either our \"length encoded\" input parser (for production) or a CSV parser (for standalone testing and debugging).\r\n\r\nWe have also previously decided that we'd like the input parsers to directly populate the dataframe to avoid any unnecessary copying.  However, this raises a number of questions:\r\n\r\n1. How will we handle control fields in the input stream sent from Java?\r\n2. The output from the analytics process needs to know the field names, but the data frame doesn't store them.  How will we deal with this?\r\n3. We have a requirement to map string inputs to numbers for the data frame, with each unique value of the string being mapped to a unique number, and then these numbers being mapped back to the original strings in the output.  Where will this mapping take place?\r\n\r\n@dimitris-athanasiou and I talked this over earlier today and this is what we came up with.\r\n\r\nConsidering the first question, of how to handle control fields:\r\n\r\n* If the input parser directly populated the data frame then the control field would have to be the first field on the line, because otherwise the input parser wouldn't realise that there was a control message until after it had already sent a row of blank values to the data frame.  Currently the Java side puts it last on the line, but this wouldn't be too hard to change.\r\n* For testing it's nice if the input file doesn't _have_ to contain a control field.\r\n\r\nThe field names are not a big problem because the `CInputParser` already provides a [method to get them](https://github.com/elastic/ml-cpp/blob/e32b8890648ee4abf81e99809c61e745d3ec7ad4/include/api/CInputParser.h#L69).\r\n\r\nFor mapping the strings to numbers we'd previously assumed the `CStringStore` class would be a good way to do this.  However, it's not really, as it maps strings to strings wrapped in shared pointers that aren't guaranteed to be unique.  We should create a new string store class that uses a Boost multi-index with hash and ordered lookups to allow mapping from string to ordinal and back again.\r\n\r\nThe existing input parsers provide each record to a callback that takes an unordered map.  This is inefficient for data frame population as the field order is critical, and the only way to get the fields in order is to iterate the field names looking up the value in the map for each name.  However, the values are copied into the map without any per-input-row memory allocations, because the same `std::string` objects are used over and over again for each row.  I think a nice way to populate the data frame is to have an option to provide the input rows as an ordered vector of strings instead of a map, but once again with the same strings being reused over an over again to avoid memory allocations.  So there would be a layer of copying of bytes, but not expensive memory allocations.  The `handleRecord` consuming the vector would then check for a control message, do the string to number mapping, and add the row to the data frame.\r\n\r\nOne final thing is that because a text field could contain a number the config must tell the analytics process which fields to parse as text and which to parse as numbers.  We could possibly have a default of analysing the first input row to find out, as this would be nice for the standalone testing case.  But for production it would be too unpredictable.  The field type config could be provided in the initial config file or as a control message."},"comment":{"url":"https://api.github.com/repos/elastic/ml-cpp/issues/comments/440658476","html_url":"https://github.com/elastic/ml-cpp/issues/315#issuecomment-440658476","issue_url":"https://api.github.com/repos/elastic/ml-cpp/issues/315","id":440658476,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MDY1ODQ3Ng==","user":{"login":"droberts195","id":7405510,"node_id":"MDQ6VXNlcjc0MDU1MTA=","avatar_url":"https://avatars0.githubusercontent.com/u/7405510?v=4","gravatar_id":"","url":"https://api.github.com/users/droberts195","html_url":"https://github.com/droberts195","followers_url":"https://api.github.com/users/droberts195/followers","following_url":"https://api.github.com/users/droberts195/following{/other_user}","gists_url":"https://api.github.com/users/droberts195/gists{/gist_id}","starred_url":"https://api.github.com/users/droberts195/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/droberts195/subscriptions","organizations_url":"https://api.github.com/users/droberts195/orgs","repos_url":"https://api.github.com/users/droberts195/repos","events_url":"https://api.github.com/users/droberts195/events{/privacy}","received_events_url":"https://api.github.com/users/droberts195/received_events","type":"User","site_admin":false},"created_at":"2018-11-21T13:16:30Z","updated_at":"2018-11-21T13:16:30Z","author_association":"CONTRIBUTOR","body":"Things have moved on since this issue was raised.\r\n\r\n> For mapping the strings to numbers\r\n\r\nThis will now be done in Java.  The C++ will work with purely numeric data frames.\r\n\r\n> a nice way to populate the data frame is to have an option to provide the input rows as an ordered vector of strings instead of a map\r\n\r\nThis was implemented in #316.\r\n\r\nI think these two pieces resolve the discussion."}}
{"actor":{"display_login":"droberts195"}}