{"action":"closed","issue":{"url":"https://api.github.com/repos/dotnet/machinelearning/issues/581","repository_url":"https://api.github.com/repos/dotnet/machinelearning","labels_url":"https://api.github.com/repos/dotnet/machinelearning/issues/581/labels{/name}","comments_url":"https://api.github.com/repos/dotnet/machinelearning/issues/581/comments","events_url":"https://api.github.com/repos/dotnet/machinelearning/issues/581/events","html_url":"https://github.com/dotnet/machinelearning/issues/581","id":344270480,"node_id":"MDU6SXNzdWUzNDQyNzA0ODA=","number":581,"title":"Three major concepts: Estimators, Transformers and Data","user":{"login":"Zruty0","id":41337831,"node_id":"MDQ6VXNlcjQxMzM3ODMx","avatar_url":"https://avatars1.githubusercontent.com/u/41337831?v=4","gravatar_id":"","url":"https://api.github.com/users/Zruty0","html_url":"https://github.com/Zruty0","followers_url":"https://api.github.com/users/Zruty0/followers","following_url":"https://api.github.com/users/Zruty0/following{/other_user}","gists_url":"https://api.github.com/users/Zruty0/gists{/gist_id}","starred_url":"https://api.github.com/users/Zruty0/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zruty0/subscriptions","organizations_url":"https://api.github.com/users/Zruty0/orgs","repos_url":"https://api.github.com/users/Zruty0/repos","events_url":"https://api.github.com/users/Zruty0/events{/privacy}","received_events_url":"https://api.github.com/users/Zruty0/received_events","type":"User","site_admin":false},"labels":[{"id":923215346,"node_id":"MDU6TGFiZWw5MjMyMTUzNDY=","url":"https://api.github.com/repos/dotnet/machinelearning/labels/api","name":"api","color":"f9d0c4","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":15,"created_at":"2018-07-25T02:39:15Z","updated_at":"2018-12-18T22:55:33Z","closed_at":"2018-12-18T22:55:32Z","author_association":"MEMBER","body":"This is still an incomplete proposal, but I played for a bit with what I had, and it looks promising to me so far.\r\n\r\nThe general idea is that we narrow our 'zoo' of components (transforms, predictors, scorers, loaders etc) down to three kinds:\r\n\r\n- The **data**. An `IDataView` with schema, like before.\r\n- The **transformer**. This is an object that can transform data and output data.\r\n```C#\r\n    public interface IDataTransformer\r\n    {\r\n        IDataView Transform(IDataView input);\r\n        ISchema GetOutputSchema(ISchema inputSchema);\r\n    }\r\n```\r\n- The **estimator**. This is the 'trainer'. The object that can 'train' a transformer using data.\r\n```C#\r\n    public interface IDataEstimator\r\n    {\r\n        IDataTransformer Fit(IDataView input);\r\n        SchemaShape GetOutputSchema(SchemaShape inputSchema);\r\n    }\r\n```\r\n\r\nObviously, a chain of transformers can itself behave as a transformer, and a chain of estimators can behave like estimators.\r\n\r\nWe also introduce a 'data reader' (and its estimator), responsible for bringing the data 'from outside' (think loaders):\r\n```C#\r\n    public interface IDataReader<TIn>\r\n    {\r\n        IDataView Read(TIn input);\r\n        ISchema GetOutputSchema();\r\n    }\r\n\r\n    public interface IDataReaderEstimator<TIn>\r\n    {\r\n        IDataReader<TIn> Fit(TIn input);\r\n        SchemaShape GetOutputSchema();\r\n    }\r\n```\r\n\r\n| Old component | New component |\r\n| ----------------- | ------------------- |\r\n| Data | Data |\r\n| Transform | Transformer |\r\n| Trainable transform (before it is trained) |  Estimator |\r\n| Trainable transform (after it is trained) |  Transformer |\r\n| Trainer | Estimator |\r\n| Predictor | **not sure yet**. I'm thinking like 'a field of the scoring transformer?' |\r\n| Scorer | Transformer |\r\n| Untrainable loader | Data reader |\r\n| Trainable loader | Estimator of data reader |\r\n\r\nI have gone through the motions of creating a 'pipeline estimator' and 'pipeline transformer' objects, which then allows me to write this code to train and test:\r\n\r\n```C#\r\n            var env = new TlcEnvironment();\r\n\r\n            var pipeline = new EstimatorPipe<IMultiStreamSource>(new MyTextLoader(env, MakeTextLoaderArgs()));\r\n            pipeline.Append(new MyConcatTransformer(env, \"Features\", \"SepalLength\", \"SepalWidth\", \"PetalLength\", \"PetalWidth\"))\r\n                    .Append(new MyNormalizer(env, \"Features\"))\r\n                    .Append(new MySdca(env));\r\n\r\n            var model = pipeline.Fit(new MultiFileSource(@\"e:\\data\\iris.txt\"));\r\n\r\n            IrisPrediction[] scoredTrainData = model.Transform(new MultiFileSource(@\"e:\\data\\iris.txt\"))\r\n                .AsEnumerable<IrisPrediction>(env, reuseRowObject: false)\r\n                .ToArray();\r\n```\r\n\r\nHere, the only catch is the 'MakeTextLoaderArgs', which is an obnoxiously long way to define the original schema of the text loader. But it is obviously subject to improvement.\r\n\r\nThe full 'playground' is available at https://github.com/Zruty0/machinelearning/tree/feature/estimators "}}
{"actor":{"display_login":"Zruty0"}}