{"action":"created","comment":{"url":"https://api.github.com/repos/logicalclocks/hops-docs/pulls/comments/244786913","pull_request_review_id":188743552,"id":244786913,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDI0NDc4NjkxMw==","diff_hunk":"@@ -0,0 +1,609 @@\n+Feature Store\n+======\n+\n+In this tutorial we cover the feature store service in Hopsworks, how it should be used, how it fits into the machine learning pipeline, and the tech-stack behind it.\n+\n+Feature Store: The Data Management Layer for Machine Learning in Hopsworks\n+------------\n+\n+The feature store is as a central place to store curated features for machine learning pipelines in Hopsworks. A feature is a measurable property of some data-sample. It could be for example an image-pixel, a word from a piece of text, the age of a person, a coordinate emitted from a sensor, or an aggregate value like the average number of purchases within the last hour. Features can come directly from tables or files or can be derived values, computed from one or more data sources.\n+\n+Features are the fuel for AI systems, as we use them to train machine learning models so that we can make predictions for feature values that we have never seen before. In this tutorial we will see best practices for transforming raw/structured data into *features* that can be included in *training datasets* for training models.\n+\n+.. _hopsworks_feature_store.png: ../../_images/feature_store/overview_new.png\n+.. figure:: ../../imgs/feature_store/overview_new.png\n+    :alt: A feature store is the interface between feature engineering and model development.\n+    :target: `hopsworks_feature_store.png`_\n+    :align: center\n+    :scale: 55 %\n+    :figclass: align-center\n+\n+    A feature store is the interface between feature engineering and model development.\n+\n+Motivation for the Feature Store\n+------------\n+\n+Machine learning systems have a tendency to assemble technical debt. Examples of technical debt in machine learning systems are:\n+\n+* There is no principled way to to access features during model serving.\n+* Features cannot easily be re-used between multiple machine learning pipelines.\n+* Data science projects work in isolation without collaboration and re-use.\n+* Features used for training and serving are inconsistent.\n+* When new data arrives, there is no way to pin down exactly which features need to be recomputed, rather the entire pipeline needs to be run to update features.\n+\n+Using a feature store is a best practice that can reduce the technical debt of machine learning work-flows. When the feature store is built up with more features, it becomes easier and cheaper to build new models as the new models can re-use features that exist in the feature store.\n+\n+How to Use the Feature Store\n+------------\n+\n+When adopting the feature store in your machine learning work-flows, you can think of it as the interface between data engineering and data science. It has two APIs, one for writing to the feature store and one for reading features. At the end of your data engineering pipeline, instead of writing features to a custom storage location, write the features to the feature store and get benefits such as automatic documentation, versioning, feature analysis, and feature sharing. As a data scientist, when you start a new machine learning project, you can search through the feature store for available features and only add new features if they do not already exist in the feature store. We encourage to reuse features as much as possible to avoid doing unnecessary work and to make features consistent between different models.\n+\n+Creating a Project on Hopsworks with The Feature Store Service Enabled\n+~~~~~~~~~~~~~~~~~~~~~~\n+\n+To create a project with the feature store service, mark the check-box available when you create a new project in Hopsworks.\n+\n+  .. _featurestore_create_project.png: ../../_images/feature_store/create_project.png\n+.. figure:: ../../imgs/feature_store/create_project.png\n+    :alt: Create a new project with the feature store service enabled.\n+    :target: `featurestore_create_project.png`_\n+    :align: center\n+    :scale: 55 %\n+    :figclass: align-center\n+\n+    Create a new project with the feature store service enabled.\n+\n+Inside the project you can find the feature registry (where all the feature store data is browsable) in the feature store page that is accessible by clicking the feature store icon on the left.\n+\n+  .. _featurestore_open_registry.png: ../../_images/feature_store/opening_feature_registry.png\n+.. figure:: ../../imgs/feature_store/opening_feature_registry.png\n+    :alt: Opening the feature store registry\n+    :target: `featurestore_open_registry.png`_\n+    :align: center\n+    :scale: 55 %\n+    :figclass: align-center\n+\n+    Opening the feature store registry in Hopsworks.\n+\n+Data Modeling in the Feature Store\n+~~~~~~~~~~~~~~~~~~~~~~\n+\n+We introduce three new concepts to our users for modeling data in the feature store.\n+\n+* The **feature** is an individual versioned and documented data column in the feature store, e.g the average rating of a customer.\n+* The **feature group** is a documented and versioned group of features stored as a Hive table. The feature group is linked to a specific Spark/Numpy/Pandas job that takes in raw data and outputs the computed features.\n+* The **training dataset** is a versioned and managed dataset of features and labels (potentially from multiple different feature groups). Training datasets are stored in HopsFS as tfrecords, parquet, csv, or tsv files.\n+\n+  .. _featurestore_concepts.png: ../../_images/feature_store/concepts.png\n+.. figure:: ../../imgs/feature_store/concepts.png\n+    :alt: Feature Store API\n+    :target: `featurestore_concepts.png`_\n+    :align: center\n+    :scale: 55 %\n+    :figclass: align-center\n+\n+    Concepts for modeling data in the feature store.\n+\n+The API\n+~~~~~~~~~~~~~~~~~~~~~~\n+\n+The feature store in Hopsworks has both a Python API and a Scala/Java API. You'll see in the code snippets below that the python API have a lot of optional arguments that are explicit in the Java API, but apart from that the APIs are identical. This section will just give an overview of the API and show examples of the most common API methods, to get a full overview please see the API docs available here: TODO and the example notebooks available here: featurestore_example_notebooks_.","path":"docs/user_guide/hopsworks/featurestore.rst","position":null,"original_position":90,"commit_id":"6c49757a77199e809620bd80ab6d21df3befde68","original_commit_id":"19d1d0113ad55327e1d0c8d24791e866540a3751","user":{"login":"Limmen","id":8254791,"node_id":"MDQ6VXNlcjgyNTQ3OTE=","avatar_url":"https://avatars0.githubusercontent.com/u/8254791?v=4","gravatar_id":"","url":"https://api.github.com/users/Limmen","html_url":"https://github.com/Limmen","followers_url":"https://api.github.com/users/Limmen/followers","following_url":"https://api.github.com/users/Limmen/following{/other_user}","gists_url":"https://api.github.com/users/Limmen/gists{/gist_id}","starred_url":"https://api.github.com/users/Limmen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Limmen/subscriptions","organizations_url":"https://api.github.com/users/Limmen/orgs","repos_url":"https://api.github.com/users/Limmen/repos","events_url":"https://api.github.com/users/Limmen/events{/privacy}","received_events_url":"https://api.github.com/users/Limmen/received_events","type":"User","site_admin":false},"body":"fixed","created_at":"2019-01-02T16:40:51Z","updated_at":"2019-01-02T16:40:52Z","html_url":"https://github.com/logicalclocks/hops-docs/pull/59#discussion_r244786913","pull_request_url":"https://api.github.com/repos/logicalclocks/hops-docs/pulls/59","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/logicalclocks/hops-docs/pulls/comments/244786913"},"html":{"href":"https://github.com/logicalclocks/hops-docs/pull/59#discussion_r244786913"},"pull_request":{"href":"https://api.github.com/repos/logicalclocks/hops-docs/pulls/59"}},"in_reply_to_id":244772641},"pull_request":{"url":"https://api.github.com/repos/logicalclocks/hops-docs/pulls/59","id":241704913,"node_id":"MDExOlB1bGxSZXF1ZXN0MjQxNzA0OTEz","html_url":"https://github.com/logicalclocks/hops-docs/pull/59","diff_url":"https://github.com/logicalclocks/hops-docs/pull/59.diff","patch_url":"https://github.com/logicalclocks/hops-docs/pull/59.patch","issue_url":"https://api.github.com/repos/logicalclocks/hops-docs/issues/59","number":59,"state":"open","locked":false,"title":"[HOPSWORKS-718] Feature Store in HopsML","user":{"login":"Limmen","id":8254791,"node_id":"MDQ6VXNlcjgyNTQ3OTE=","avatar_url":"https://avatars0.githubusercontent.com/u/8254791?v=4","gravatar_id":"","url":"https://api.github.com/users/Limmen","html_url":"https://github.com/Limmen","followers_url":"https://api.github.com/users/Limmen/followers","following_url":"https://api.github.com/users/Limmen/following{/other_user}","gists_url":"https://api.github.com/users/Limmen/gists{/gist_id}","starred_url":"https://api.github.com/users/Limmen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Limmen/subscriptions","organizations_url":"https://api.github.com/users/Limmen/orgs","repos_url":"https://api.github.com/users/Limmen/repos","events_url":"https://api.github.com/users/Limmen/events{/privacy}","received_events_url":"https://api.github.com/users/Limmen/received_events","type":"User","site_admin":false},"body":"- Adds documentation for the feature store in HopsML","created_at":"2019-01-02T09:46:29Z","updated_at":"2019-01-02T16:40:52Z","closed_at":null,"merged_at":null,"merge_commit_sha":"e0441b266ebe292e191efed5eace8eba1c989962","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/logicalclocks/hops-docs/pulls/59/commits","review_comments_url":"https://api.github.com/repos/logicalclocks/hops-docs/pulls/59/comments","review_comment_url":"https://api.github.com/repos/logicalclocks/hops-docs/pulls/comments{/number}","comments_url":"https://api.github.com/repos/logicalclocks/hops-docs/issues/59/comments","statuses_url":"https://api.github.com/repos/logicalclocks/hops-docs/statuses/6c49757a77199e809620bd80ab6d21df3befde68","head":{"label":"Limmen:HOPSWORKS-718","ref":"HOPSWORKS-718","sha":"6c49757a77199e809620bd80ab6d21df3befde68","user":{"login":"Limmen","id":8254791,"node_id":"MDQ6VXNlcjgyNTQ3OTE=","avatar_url":"https://avatars0.githubusercontent.com/u/8254791?v=4","gravatar_id":"","url":"https://api.github.com/users/Limmen","html_url":"https://github.com/Limmen","followers_url":"https://api.github.com/users/Limmen/followers","following_url":"https://api.github.com/users/Limmen/following{/other_user}","gists_url":"https://api.github.com/users/Limmen/gists{/gist_id}","starred_url":"https://api.github.com/users/Limmen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Limmen/subscriptions","organizations_url":"https://api.github.com/users/Limmen/orgs","repos_url":"https://api.github.com/users/Limmen/repos","events_url":"https://api.github.com/users/Limmen/events{/privacy}","received_events_url":"https://api.github.com/users/Limmen/received_events","type":"User","site_admin":false},"repo":{"id":151597669,"node_id":"MDEwOlJlcG9zaXRvcnkxNTE1OTc2Njk=","name":"hops-docs","full_name":"Limmen/hops-docs","private":false,"owner":{"login":"Limmen","id":8254791,"node_id":"MDQ6VXNlcjgyNTQ3OTE=","avatar_url":"https://avatars0.githubusercontent.com/u/8254791?v=4","gravatar_id":"","url":"https://api.github.com/users/Limmen","html_url":"https://github.com/Limmen","followers_url":"https://api.github.com/users/Limmen/followers","following_url":"https://api.github.com/users/Limmen/following{/other_user}","gists_url":"https://api.github.com/users/Limmen/gists{/gist_id}","starred_url":"https://api.github.com/users/Limmen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Limmen/subscriptions","organizations_url":"https://api.github.com/users/Limmen/orgs","repos_url":"https://api.github.com/users/Limmen/repos","events_url":"https://api.github.com/users/Limmen/events{/privacy}","received_events_url":"https://api.github.com/users/Limmen/received_events","type":"User","site_admin":false},"html_url":"https://github.com/Limmen/hops-docs","description":"Documentation for Hopsworks, HopsFS, and HopsYARN, as well as installation.","fork":true,"url":"https://api.github.com/repos/Limmen/hops-docs","forks_url":"https://api.github.com/repos/Limmen/hops-docs/forks","keys_url":"https://api.github.com/repos/Limmen/hops-docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Limmen/hops-docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Limmen/hops-docs/teams","hooks_url":"https://api.github.com/repos/Limmen/hops-docs/hooks","issue_events_url":"https://api.github.com/repos/Limmen/hops-docs/issues/events{/number}","events_url":"https://api.github.com/repos/Limmen/hops-docs/events","assignees_url":"https://api.github.com/repos/Limmen/hops-docs/assignees{/user}","branches_url":"https://api.github.com/repos/Limmen/hops-docs/branches{/branch}","tags_url":"https://api.github.com/repos/Limmen/hops-docs/tags","blobs_url":"https://api.github.com/repos/Limmen/hops-docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Limmen/hops-docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Limmen/hops-docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/Limmen/hops-docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Limmen/hops-docs/statuses/{sha}","languages_url":"https://api.github.com/repos/Limmen/hops-docs/languages","stargazers_url":"https://api.github.com/repos/Limmen/hops-docs/stargazers","contributors_url":"https://api.github.com/repos/Limmen/hops-docs/contributors","subscribers_url":"https://api.github.com/repos/Limmen/hops-docs/subscribers","subscription_url":"https://api.github.com/repos/Limmen/hops-docs/subscription","commits_url":"https://api.github.com/repos/Limmen/hops-docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/Limmen/hops-docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/Limmen/hops-docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/Limmen/hops-docs/issues/comments{/number}","contents_url":"https://api.github.com/repos/Limmen/hops-docs/contents/{+path}","compare_url":"https://api.github.com/repos/Limmen/hops-docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Limmen/hops-docs/merges","archive_url":"https://api.github.com/repos/Limmen/hops-docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Limmen/hops-docs/downloads","issues_url":"https://api.github.com/repos/Limmen/hops-docs/issues{/number}","pulls_url":"https://api.github.com/repos/Limmen/hops-docs/pulls{/number}","milestones_url":"https://api.github.com/repos/Limmen/hops-docs/milestones{/number}","notifications_url":"https://api.github.com/repos/Limmen/hops-docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Limmen/hops-docs/labels{/name}","releases_url":"https://api.github.com/repos/Limmen/hops-docs/releases{/id}","deployments_url":"https://api.github.com/repos/Limmen/hops-docs/deployments","created_at":"2018-10-04T15:53:05Z","updated_at":"2018-10-04T15:53:08Z","pushed_at":"2019-01-02T16:40:12Z","git_url":"git://github.com/Limmen/hops-docs.git","ssh_url":"git@github.com:Limmen/hops-docs.git","clone_url":"https://github.com/Limmen/hops-docs.git","svn_url":"https://github.com/Limmen/hops-docs","homepage":null,"size":13653,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"agpl-3.0","name":"GNU Affero General Public License v3.0","spdx_id":"AGPL-3.0","url":"https://api.github.com/licenses/agpl-3.0","node_id":"MDc6TGljZW5zZTE="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"logicalclocks:master","ref":"master","sha":"bc3934754cff29e3627d06fac9a4d5bc41de168a","user":{"login":"logicalclocks","id":26795543,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI2Nzk1NTQz","avatar_url":"https://avatars3.githubusercontent.com/u/26795543?v=4","gravatar_id":"","url":"https://api.github.com/users/logicalclocks","html_url":"https://github.com/logicalclocks","followers_url":"https://api.github.com/users/logicalclocks/followers","following_url":"https://api.github.com/users/logicalclocks/following{/other_user}","gists_url":"https://api.github.com/users/logicalclocks/gists{/gist_id}","starred_url":"https://api.github.com/users/logicalclocks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/logicalclocks/subscriptions","organizations_url":"https://api.github.com/users/logicalclocks/orgs","repos_url":"https://api.github.com/users/logicalclocks/repos","events_url":"https://api.github.com/users/logicalclocks/events{/privacy}","received_events_url":"https://api.github.com/users/logicalclocks/received_events","type":"Organization","site_admin":false},"repo":{"id":44493382,"node_id":"MDEwOlJlcG9zaXRvcnk0NDQ5MzM4Mg==","name":"hops-docs","full_name":"logicalclocks/hops-docs","private":false,"owner":{"login":"logicalclocks","id":26795543,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI2Nzk1NTQz","avatar_url":"https://avatars3.githubusercontent.com/u/26795543?v=4","gravatar_id":"","url":"https://api.github.com/users/logicalclocks","html_url":"https://github.com/logicalclocks","followers_url":"https://api.github.com/users/logicalclocks/followers","following_url":"https://api.github.com/users/logicalclocks/following{/other_user}","gists_url":"https://api.github.com/users/logicalclocks/gists{/gist_id}","starred_url":"https://api.github.com/users/logicalclocks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/logicalclocks/subscriptions","organizations_url":"https://api.github.com/users/logicalclocks/orgs","repos_url":"https://api.github.com/users/logicalclocks/repos","events_url":"https://api.github.com/users/logicalclocks/events{/privacy}","received_events_url":"https://api.github.com/users/logicalclocks/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/logicalclocks/hops-docs","description":"Documentation for Hopsworks and Hops","fork":false,"url":"https://api.github.com/repos/logicalclocks/hops-docs","forks_url":"https://api.github.com/repos/logicalclocks/hops-docs/forks","keys_url":"https://api.github.com/repos/logicalclocks/hops-docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/logicalclocks/hops-docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/logicalclocks/hops-docs/teams","hooks_url":"https://api.github.com/repos/logicalclocks/hops-docs/hooks","issue_events_url":"https://api.github.com/repos/logicalclocks/hops-docs/issues/events{/number}","events_url":"https://api.github.com/repos/logicalclocks/hops-docs/events","assignees_url":"https://api.github.com/repos/logicalclocks/hops-docs/assignees{/user}","branches_url":"https://api.github.com/repos/logicalclocks/hops-docs/branches{/branch}","tags_url":"https://api.github.com/repos/logicalclocks/hops-docs/tags","blobs_url":"https://api.github.com/repos/logicalclocks/hops-docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/logicalclocks/hops-docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/logicalclocks/hops-docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/logicalclocks/hops-docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/logicalclocks/hops-docs/statuses/{sha}","languages_url":"https://api.github.com/repos/logicalclocks/hops-docs/languages","stargazers_url":"https://api.github.com/repos/logicalclocks/hops-docs/stargazers","contributors_url":"https://api.github.com/repos/logicalclocks/hops-docs/contributors","subscribers_url":"https://api.github.com/repos/logicalclocks/hops-docs/subscribers","subscription_url":"https://api.github.com/repos/logicalclocks/hops-docs/subscription","commits_url":"https://api.github.com/repos/logicalclocks/hops-docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/logicalclocks/hops-docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/logicalclocks/hops-docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/logicalclocks/hops-docs/issues/comments{/number}","contents_url":"https://api.github.com/repos/logicalclocks/hops-docs/contents/{+path}","compare_url":"https://api.github.com/repos/logicalclocks/hops-docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/logicalclocks/hops-docs/merges","archive_url":"https://api.github.com/repos/logicalclocks/hops-docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/logicalclocks/hops-docs/downloads","issues_url":"https://api.github.com/repos/logicalclocks/hops-docs/issues{/number}","pulls_url":"https://api.github.com/repos/logicalclocks/hops-docs/pulls{/number}","milestones_url":"https://api.github.com/repos/logicalclocks/hops-docs/milestones{/number}","notifications_url":"https://api.github.com/repos/logicalclocks/hops-docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/logicalclocks/hops-docs/labels{/name}","releases_url":"https://api.github.com/repos/logicalclocks/hops-docs/releases{/id}","deployments_url":"https://api.github.com/repos/logicalclocks/hops-docs/deployments","created_at":"2015-10-18T19:42:31Z","updated_at":"2019-01-02T16:25:59Z","pushed_at":"2019-01-02T16:40:14Z","git_url":"git://github.com/logicalclocks/hops-docs.git","ssh_url":"git@github.com:logicalclocks/hops-docs.git","clone_url":"https://github.com/logicalclocks/hops-docs.git","svn_url":"https://github.com/logicalclocks/hops-docs","homepage":"","size":12109,"stargazers_count":2,"watchers_count":2,"language":null,"has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":11,"mirror_url":null,"archived":false,"open_issues_count":1,"license":{"key":"agpl-3.0","name":"GNU Affero General Public License v3.0","spdx_id":"AGPL-3.0","url":"https://api.github.com/licenses/agpl-3.0","node_id":"MDc6TGljZW5zZTE="},"forks":11,"open_issues":1,"watchers":2,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/logicalclocks/hops-docs/pulls/59"},"html":{"href":"https://github.com/logicalclocks/hops-docs/pull/59"},"issue":{"href":"https://api.github.com/repos/logicalclocks/hops-docs/issues/59"},"comments":{"href":"https://api.github.com/repos/logicalclocks/hops-docs/issues/59/comments"},"review_comments":{"href":"https://api.github.com/repos/logicalclocks/hops-docs/pulls/59/comments"},"review_comment":{"href":"https://api.github.com/repos/logicalclocks/hops-docs/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/logicalclocks/hops-docs/pulls/59/commits"},"statuses":{"href":"https://api.github.com/repos/logicalclocks/hops-docs/statuses/6c49757a77199e809620bd80ab6d21df3befde68"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"Limmen"}}