{"action":"created","comment":{"url":"https://api.github.com/repos/juttle/juttle/pulls/comments/54819778","id":54819778,"diff_hunk":"@@ -6,9 +6,65 @@ NOTE: This API is currently under active development and is highly likely to cha\n \n __TOC__\n \n+## Overview\n+\n+An Adapter is the interface between the juttle runtime and a backend data source. The backend consists of a set of data points. Each data point contains a number of fields. In most cases, each data point also contains a timestamp.\n+\n+For reads, the adapter's job is to interpret the options included in the juttle `read` command into a set of matching data points, construct json objects representing those data points, and pass them to the juttle runtime. For writes, the adapter's job is to take the output of programs and \"save\" that output by writing data points to the backend.\n+\n+More sophisticated adapters work together with the juttle optimizer to push aggregation operations directly into the backend. For example, to count the number of data points in a given time period you could simply fetch all the data points and have the juttle program perform the counting. However, if supported by the backend it would be more efficient to count the number of data points directly in the backend and return the count instead.\n+\n+## Writing an Adapter\n+\n+Here are the main steps involved in writing a new adapter:\n+\n+1. Create a javascript module that is require()d when a juttle program executes a `read <adapter>` or `write <adapter>`.","path":"docs/adapters/adapter_api.md","position":16,"original_position":16,"commit_id":"172b9ab092fdb22376bbe82d5be2f2dd53962114","original_commit_id":"172b9ab092fdb22376bbe82d5be2f2dd53962114","user":{"login":"davidvgalbraith","id":2757844,"avatar_url":"https://avatars.githubusercontent.com/u/2757844?v=3","gravatar_id":"","url":"https://api.github.com/users/davidvgalbraith","html_url":"https://github.com/davidvgalbraith","followers_url":"https://api.github.com/users/davidvgalbraith/followers","following_url":"https://api.github.com/users/davidvgalbraith/following{/other_user}","gists_url":"https://api.github.com/users/davidvgalbraith/gists{/gist_id}","starred_url":"https://api.github.com/users/davidvgalbraith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidvgalbraith/subscriptions","organizations_url":"https://api.github.com/users/davidvgalbraith/orgs","repos_url":"https://api.github.com/users/davidvgalbraith/repos","events_url":"https://api.github.com/users/davidvgalbraith/events{/privacy}","received_events_url":"https://api.github.com/users/davidvgalbraith/received_events","type":"User","site_admin":false},"body":"The \"that is require()d when a juttle program executes a `read <adapter>` or `write <adapter>`.\" is pretty intense for a step 1 and it's not clear how to do that. How about just `Create a javascript module`?","created_at":"2016-03-03T00:47:41Z","updated_at":"2016-03-03T00:47:41Z","html_url":"https://github.com/juttle/juttle/pull/505#discussion_r54819778","pull_request_url":"https://api.github.com/repos/juttle/juttle/pulls/505","_links":{"self":{"href":"https://api.github.com/repos/juttle/juttle/pulls/comments/54819778"},"html":{"href":"https://github.com/juttle/juttle/pull/505#discussion_r54819778"},"pull_request":{"href":"https://api.github.com/repos/juttle/juttle/pulls/505"}}},"pull_request":{"url":"https://api.github.com/repos/juttle/juttle/pulls/505","id":61444160,"html_url":"https://github.com/juttle/juttle/pull/505","diff_url":"https://github.com/juttle/juttle/pull/505.diff","patch_url":"https://github.com/juttle/juttle/pull/505.patch","issue_url":"https://api.github.com/repos/juttle/juttle/issues/505","number":505,"state":"open","locked":false,"title":"Add more detail on how to write adapters.","user":{"login":"mstemm","id":13422831,"avatar_url":"https://avatars.githubusercontent.com/u/13422831?v=3","gravatar_id":"","url":"https://api.github.com/users/mstemm","html_url":"https://github.com/mstemm","followers_url":"https://api.github.com/users/mstemm/followers","following_url":"https://api.github.com/users/mstemm/following{/other_user}","gists_url":"https://api.github.com/users/mstemm/gists{/gist_id}","starred_url":"https://api.github.com/users/mstemm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mstemm/subscriptions","organizations_url":"https://api.github.com/users/mstemm/orgs","repos_url":"https://api.github.com/users/mstemm/repos","events_url":"https://api.github.com/users/mstemm/events{/privacy}","received_events_url":"https://api.github.com/users/mstemm/received_events","type":"User","site_admin":false},"body":"Take most of the general discussion from the gmail adapter docs at\r\nhttps://github.com/juttle/juttle-gmail-adapter/blob/master/docs/adapter_impl_notes.md\r\nand add them to this doc. Specifically, added the following:\r\n\r\n - How juttle loads an adapter\r\n - General discussion of live/historical and how that relates to the data read/written by an adapter.\r\n - Notes on configuration.\r\n - Options supported by many read commands.\r\n - Notes on the filter expression and how its parsed version is made available to the adapter.\r\n - Notes on write and expectations for write (don't block event loop) and eof (return a promise that resolves when everything is written).\r\n - Logging\r\n - Returning errors.\r\n\r\nOnce this is merged I'll change the gmail adapter docs to take out the\r\ngeneral stuff and tailor it more closely to the specific\r\nchoices/options used by the gmail adapter.\r\n\r\n@demmer @dmehra @rlgomes ","created_at":"2016-03-02T18:24:58Z","updated_at":"2016-03-03T00:47:41Z","closed_at":null,"merged_at":null,"merge_commit_sha":"5510e165fd767575c17dc92e0c1210bcf0cea656","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/juttle/juttle/pulls/505/commits","review_comments_url":"https://api.github.com/repos/juttle/juttle/pulls/505/comments","review_comment_url":"https://api.github.com/repos/juttle/juttle/pulls/comments{/number}","comments_url":"https://api.github.com/repos/juttle/juttle/issues/505/comments","statuses_url":"https://api.github.com/repos/juttle/juttle/statuses/172b9ab092fdb22376bbe82d5be2f2dd53962114","head":{"label":"juttle:improved-adapter-api-doc","ref":"improved-adapter-api-doc","sha":"172b9ab092fdb22376bbe82d5be2f2dd53962114","user":{"login":"juttle","id":8988633,"avatar_url":"https://avatars.githubusercontent.com/u/8988633?v=3","gravatar_id":"","url":"https://api.github.com/users/juttle","html_url":"https://github.com/juttle","followers_url":"https://api.github.com/users/juttle/followers","following_url":"https://api.github.com/users/juttle/following{/other_user}","gists_url":"https://api.github.com/users/juttle/gists{/gist_id}","starred_url":"https://api.github.com/users/juttle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juttle/subscriptions","organizations_url":"https://api.github.com/users/juttle/orgs","repos_url":"https://api.github.com/users/juttle/repos","events_url":"https://api.github.com/users/juttle/events{/privacy}","received_events_url":"https://api.github.com/users/juttle/received_events","type":"Organization","site_admin":false},"repo":{"id":48260003,"name":"juttle","full_name":"juttle/juttle","owner":{"login":"juttle","id":8988633,"avatar_url":"https://avatars.githubusercontent.com/u/8988633?v=3","gravatar_id":"","url":"https://api.github.com/users/juttle","html_url":"https://github.com/juttle","followers_url":"https://api.github.com/users/juttle/followers","following_url":"https://api.github.com/users/juttle/following{/other_user}","gists_url":"https://api.github.com/users/juttle/gists{/gist_id}","starred_url":"https://api.github.com/users/juttle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juttle/subscriptions","organizations_url":"https://api.github.com/users/juttle/orgs","repos_url":"https://api.github.com/users/juttle/repos","events_url":"https://api.github.com/users/juttle/events{/privacy}","received_events_url":"https://api.github.com/users/juttle/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/juttle/juttle","description":"Juttle dataflow programming language","fork":false,"url":"https://api.github.com/repos/juttle/juttle","forks_url":"https://api.github.com/repos/juttle/juttle/forks","keys_url":"https://api.github.com/repos/juttle/juttle/keys{/key_id}","collaborators_url":"https://api.github.com/repos/juttle/juttle/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/juttle/juttle/teams","hooks_url":"https://api.github.com/repos/juttle/juttle/hooks","issue_events_url":"https://api.github.com/repos/juttle/juttle/issues/events{/number}","events_url":"https://api.github.com/repos/juttle/juttle/events","assignees_url":"https://api.github.com/repos/juttle/juttle/assignees{/user}","branches_url":"https://api.github.com/repos/juttle/juttle/branches{/branch}","tags_url":"https://api.github.com/repos/juttle/juttle/tags","blobs_url":"https://api.github.com/repos/juttle/juttle/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/juttle/juttle/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/juttle/juttle/git/refs{/sha}","trees_url":"https://api.github.com/repos/juttle/juttle/git/trees{/sha}","statuses_url":"https://api.github.com/repos/juttle/juttle/statuses/{sha}","languages_url":"https://api.github.com/repos/juttle/juttle/languages","stargazers_url":"https://api.github.com/repos/juttle/juttle/stargazers","contributors_url":"https://api.github.com/repos/juttle/juttle/contributors","subscribers_url":"https://api.github.com/repos/juttle/juttle/subscribers","subscription_url":"https://api.github.com/repos/juttle/juttle/subscription","commits_url":"https://api.github.com/repos/juttle/juttle/commits{/sha}","git_commits_url":"https://api.github.com/repos/juttle/juttle/git/commits{/sha}","comments_url":"https://api.github.com/repos/juttle/juttle/comments{/number}","issue_comment_url":"https://api.github.com/repos/juttle/juttle/issues/comments{/number}","contents_url":"https://api.github.com/repos/juttle/juttle/contents/{+path}","compare_url":"https://api.github.com/repos/juttle/juttle/compare/{base}...{head}","merges_url":"https://api.github.com/repos/juttle/juttle/merges","archive_url":"https://api.github.com/repos/juttle/juttle/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/juttle/juttle/downloads","issues_url":"https://api.github.com/repos/juttle/juttle/issues{/number}","pulls_url":"https://api.github.com/repos/juttle/juttle/pulls{/number}","milestones_url":"https://api.github.com/repos/juttle/juttle/milestones{/number}","notifications_url":"https://api.github.com/repos/juttle/juttle/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/juttle/juttle/labels{/name}","releases_url":"https://api.github.com/repos/juttle/juttle/releases{/id}","deployments_url":"https://api.github.com/repos/juttle/juttle/deployments","created_at":"2015-12-18T22:50:40Z","updated_at":"2016-03-02T19:53:46Z","pushed_at":"2016-03-03T00:20:20Z","git_url":"git://github.com/juttle/juttle.git","ssh_url":"git@github.com:juttle/juttle.git","clone_url":"https://github.com/juttle/juttle.git","svn_url":"https://github.com/juttle/juttle","homepage":"","size":4690,"stargazers_count":49,"watchers_count":49,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":7,"mirror_url":null,"open_issues_count":99,"forks":7,"open_issues":99,"watchers":49,"default_branch":"master"}},"base":{"label":"juttle:master","ref":"master","sha":"7f2255a99c45b5d509bb82d3464cca18c21ce0eb","user":{"login":"juttle","id":8988633,"avatar_url":"https://avatars.githubusercontent.com/u/8988633?v=3","gravatar_id":"","url":"https://api.github.com/users/juttle","html_url":"https://github.com/juttle","followers_url":"https://api.github.com/users/juttle/followers","following_url":"https://api.github.com/users/juttle/following{/other_user}","gists_url":"https://api.github.com/users/juttle/gists{/gist_id}","starred_url":"https://api.github.com/users/juttle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juttle/subscriptions","organizations_url":"https://api.github.com/users/juttle/orgs","repos_url":"https://api.github.com/users/juttle/repos","events_url":"https://api.github.com/users/juttle/events{/privacy}","received_events_url":"https://api.github.com/users/juttle/received_events","type":"Organization","site_admin":false},"repo":{"id":48260003,"name":"juttle","full_name":"juttle/juttle","owner":{"login":"juttle","id":8988633,"avatar_url":"https://avatars.githubusercontent.com/u/8988633?v=3","gravatar_id":"","url":"https://api.github.com/users/juttle","html_url":"https://github.com/juttle","followers_url":"https://api.github.com/users/juttle/followers","following_url":"https://api.github.com/users/juttle/following{/other_user}","gists_url":"https://api.github.com/users/juttle/gists{/gist_id}","starred_url":"https://api.github.com/users/juttle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juttle/subscriptions","organizations_url":"https://api.github.com/users/juttle/orgs","repos_url":"https://api.github.com/users/juttle/repos","events_url":"https://api.github.com/users/juttle/events{/privacy}","received_events_url":"https://api.github.com/users/juttle/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/juttle/juttle","description":"Juttle dataflow programming language","fork":false,"url":"https://api.github.com/repos/juttle/juttle","forks_url":"https://api.github.com/repos/juttle/juttle/forks","keys_url":"https://api.github.com/repos/juttle/juttle/keys{/key_id}","collaborators_url":"https://api.github.com/repos/juttle/juttle/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/juttle/juttle/teams","hooks_url":"https://api.github.com/repos/juttle/juttle/hooks","issue_events_url":"https://api.github.com/repos/juttle/juttle/issues/events{/number}","events_url":"https://api.github.com/repos/juttle/juttle/events","assignees_url":"https://api.github.com/repos/juttle/juttle/assignees{/user}","branches_url":"https://api.github.com/repos/juttle/juttle/branches{/branch}","tags_url":"https://api.github.com/repos/juttle/juttle/tags","blobs_url":"https://api.github.com/repos/juttle/juttle/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/juttle/juttle/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/juttle/juttle/git/refs{/sha}","trees_url":"https://api.github.com/repos/juttle/juttle/git/trees{/sha}","statuses_url":"https://api.github.com/repos/juttle/juttle/statuses/{sha}","languages_url":"https://api.github.com/repos/juttle/juttle/languages","stargazers_url":"https://api.github.com/repos/juttle/juttle/stargazers","contributors_url":"https://api.github.com/repos/juttle/juttle/contributors","subscribers_url":"https://api.github.com/repos/juttle/juttle/subscribers","subscription_url":"https://api.github.com/repos/juttle/juttle/subscription","commits_url":"https://api.github.com/repos/juttle/juttle/commits{/sha}","git_commits_url":"https://api.github.com/repos/juttle/juttle/git/commits{/sha}","comments_url":"https://api.github.com/repos/juttle/juttle/comments{/number}","issue_comment_url":"https://api.github.com/repos/juttle/juttle/issues/comments{/number}","contents_url":"https://api.github.com/repos/juttle/juttle/contents/{+path}","compare_url":"https://api.github.com/repos/juttle/juttle/compare/{base}...{head}","merges_url":"https://api.github.com/repos/juttle/juttle/merges","archive_url":"https://api.github.com/repos/juttle/juttle/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/juttle/juttle/downloads","issues_url":"https://api.github.com/repos/juttle/juttle/issues{/number}","pulls_url":"https://api.github.com/repos/juttle/juttle/pulls{/number}","milestones_url":"https://api.github.com/repos/juttle/juttle/milestones{/number}","notifications_url":"https://api.github.com/repos/juttle/juttle/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/juttle/juttle/labels{/name}","releases_url":"https://api.github.com/repos/juttle/juttle/releases{/id}","deployments_url":"https://api.github.com/repos/juttle/juttle/deployments","created_at":"2015-12-18T22:50:40Z","updated_at":"2016-03-02T19:53:46Z","pushed_at":"2016-03-03T00:20:20Z","git_url":"git://github.com/juttle/juttle.git","ssh_url":"git@github.com:juttle/juttle.git","clone_url":"https://github.com/juttle/juttle.git","svn_url":"https://github.com/juttle/juttle","homepage":"","size":4690,"stargazers_count":49,"watchers_count":49,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":7,"mirror_url":null,"open_issues_count":99,"forks":7,"open_issues":99,"watchers":49,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/juttle/juttle/pulls/505"},"html":{"href":"https://github.com/juttle/juttle/pull/505"},"issue":{"href":"https://api.github.com/repos/juttle/juttle/issues/505"},"comments":{"href":"https://api.github.com/repos/juttle/juttle/issues/505/comments"},"review_comments":{"href":"https://api.github.com/repos/juttle/juttle/pulls/505/comments"},"review_comment":{"href":"https://api.github.com/repos/juttle/juttle/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/juttle/juttle/pulls/505/commits"},"statuses":{"href":"https://api.github.com/repos/juttle/juttle/statuses/172b9ab092fdb22376bbe82d5be2f2dd53962114"}}}}