{"action":"created","issue":{"url":"https://api.github.com/repos/deepchem/deepchem/issues/1039","repository_url":"https://api.github.com/repos/deepchem/deepchem","labels_url":"https://api.github.com/repos/deepchem/deepchem/issues/1039/labels{/name}","comments_url":"https://api.github.com/repos/deepchem/deepchem/issues/1039/comments","events_url":"https://api.github.com/repos/deepchem/deepchem/issues/1039/events","html_url":"https://github.com/deepchem/deepchem/issues/1039","id":290742716,"number":1039,"title":"Integrating tf.data and dc.data","user":{"login":"rbharath","id":1057676,"avatar_url":"https://avatars3.githubusercontent.com/u/1057676?v=4","gravatar_id":"","url":"https://api.github.com/users/rbharath","html_url":"https://github.com/rbharath","followers_url":"https://api.github.com/users/rbharath/followers","following_url":"https://api.github.com/users/rbharath/following{/other_user}","gists_url":"https://api.github.com/users/rbharath/gists{/gist_id}","starred_url":"https://api.github.com/users/rbharath/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rbharath/subscriptions","organizations_url":"https://api.github.com/users/rbharath/orgs","repos_url":"https://api.github.com/users/rbharath/repos","events_url":"https://api.github.com/users/rbharath/events{/privacy}","received_events_url":"https://api.github.com/users/rbharath/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-01-23T08:02:08Z","updated_at":"2018-02-14T23:55:45Z","closed_at":null,"author_association":"OWNER","body":"Continuing our architectural discussions, it's starting to become clear we should integrate dc.data and tf.data in some manner. dc.data is one of the hackier parts of DeepChem and has caused us a good number of scaling and portability headaches. Swapping out this code for optimized tf defaults seems generally useful.\r\n\r\nAt the same time, we need to plan a transition that doesn't entirely break backwards compatibility. Let's use this thread to brainstorm possible implementation paths."},"comment":{"url":"https://api.github.com/repos/deepchem/deepchem/issues/comments/365784623","html_url":"https://github.com/deepchem/deepchem/issues/1039#issuecomment-365784623","issue_url":"https://api.github.com/repos/deepchem/deepchem/issues/1039","id":365784623,"user":{"login":"peastman","id":4379786,"avatar_url":"https://avatars3.githubusercontent.com/u/4379786?v=4","gravatar_id":"","url":"https://api.github.com/users/peastman","html_url":"https://github.com/peastman","followers_url":"https://api.github.com/users/peastman/followers","following_url":"https://api.github.com/users/peastman/following{/other_user}","gists_url":"https://api.github.com/users/peastman/gists{/gist_id}","starred_url":"https://api.github.com/users/peastman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peastman/subscriptions","organizations_url":"https://api.github.com/users/peastman/orgs","repos_url":"https://api.github.com/users/peastman/repos","events_url":"https://api.github.com/users/peastman/events{/privacy}","received_events_url":"https://api.github.com/users/peastman/received_events","type":"User","site_admin":false},"created_at":"2018-02-14T23:55:45Z","updated_at":"2018-02-14T23:55:45Z","author_association":"CONTRIBUTOR","body":"tf.data is very flexible but also awkward to use.  There's nothing nearly as simple as just passing a Dataset object to a model.  Instead, every time you use a dataset you have to write an \"input function\" that's responsible for gathering up the data and packaging it in whatever way is required.\r\n\r\nI think this feature request involves two pieces:\r\n\r\n1. Allow a DeepChem Dataset to be used with models designed for use with tf.data.  I don't think this should be hard, especially given that you're expected to write adapter code anyway.  A simple implementation would be to add a `make_iterator()` method to Dataset.  It would return a tf.data.Iterator, which exposes tensors for reading the data.  I'll try implementing this.\r\n\r\n2. Allow TensorGraph models to work with tf.data data sources.  This will require more thought.  Possibly we want to redesign TensorGraph to be a subclass of tf.estimator.Estimator.  Or we might create a way to wrap a TensorGraph in an Estimator, similar to how `tf.keras.estimator.model_to_estimator()` works."}}
{"actor":{"display_login":"peastman"}}