{"action":"created","comment":{"url":"https://api.github.com/repos/influxdata/influxdb/pulls/comments/52826089","id":52826089,"diff_hunk":"@@ -60,66 +60,66 @@ The last section is the footer that stores the offset of the start of the index.\n \n # File System Layout\n \n-The file system is organized a directory per shard where each shard is integer number.  Within the shard dir exists a set of other directories and files:\n+The file system is organized a directory per shard where each shard is an integer number.  Within each shard directory, there is a set of other directories and files:\n \n-* wal Dir - Contains a set numerically increasing files WAL segment files name ######.wal.  The wal dir will be a separate location from the TSM data files so that different types can be used if necessary.\n-* TSM files - A set of numerically increasing TSM files containing compressed series data.\n-* Tombstone files - Files named after the corresponding TSM file as #####.tombstone.  These contain measurement and series keys that have been deleted.  These are removed during compactions.\n+* a wal directory - contains a set numerically increasing files WAL segment files named #####.wal.  The wal directory is separate from the directory containing the TSM files so that different types can be used if necessary.\n+* .tsm files - a set of numerically increasing TSM files containing compressed series data.\n+* .tombstone files - files named after the corresponding TSM file as #####.tombstone.  These contain measurement and series keys that have been deleted.  These files are removed during compactions.\n \n # Data Flow\n \n Writes are appended to the current WAL segment and are also added to the Cache.  Each WAL segment is size bounded and rolls-over to a new file after it fills up.  The cache is also size bounded and older entries are evicted as new entries are added to maintain the size.  The WAL and Cache are separate entities and do not interact with each other.  The Engine coordinates the writes to both.\n \n-When WAL segments fill up and closed, the Compactor reads the WAL entries and combines then with one or more existing TSM files.  This process runs continously until all WAL files are compacted and there is a minimum number of TSM files.  As each TSM file is completed, it is loaded and referenced by the FileStore.\n+When WAL segments fill up and have been closed, the Compactor reads the WAL entries and combines them with one or more existing TSM files.  This process runs continuously until all WAL files are compacted and there is a minimum number of TSM files.  As each TSM file is completed, it is loaded and referenced by the FileStore.\n \n-Queries are executed by constructing Cursors for keys.  The Cursors iterate of slices of Values.  When the current Values are exhausted, a cursor requests a the next set of Values from the Engine.  The Engine returns a slice of Values by querying the FileStore and Cache.  The Values in the Cache are overlayed on top of the values returned from the FileStore.  The FileStore reads and decodes blocks of Values according to the index for the file.\n+Queries are executed by constructing Cursors for keys.  The Cursors iterate over slices of Values.  When the current Values are exhausted, a Cursor requests the next set of Values from the Engine.  The Engine returns a slice of Values by querying the FileStore and Cache.  The Values in the Cache are overlaid on top of the values returned from the FileStore.  The FileStore reads and decodes blocks of Values according to the index for the file.\n \n Updates (writing a newer value for a point that already exists) occur as normal writes.  Since cached values overwrite existing values, newer writes take precedence.\n \n-Deletes occur by writing a delete entry for the measurement or series to the WAL and then update the Cache and FileStore.  The Cache evicts all relevant entries.  The FileStore writes a tombstone file for each TSM file that contains relevent data.  These tombstone files are used at startup time to ignore blocks as well as during compactions to remove deleted entries.\n+Deletes occur by writing a delete entry for the measurement or series to the WAL and then updating the Cache and FileStore.  The Cache evicts all relevant entries.  The FileStore writes a tombstone file for each TSM file that contains relevant data.  These tombstone files are used at startup time to ignore blocks as well as during compactions to remove deleted entries.\n \n # Compactions\n \n-Compactions are a serial and continously running process that iteratively optimizes the storage for queries.  Specifically, it does the following:\n+Compactions are a serial and continuously running process that iteratively optimizes the storage for queries.  Specifically, it does the following:\n \n * Converts closed WAL files into TSM files and removes the closed WAL files\n * Combines smaller TSM files into larger ones to improve compression ratios\n * Rewrites existing files that contain series data that has been deleted\n * Rewrites existing files that contain writes with more recent data to ensure a point exists in only one TSM file.\n \n-The compaction algorithm is continously running and always selects files to compact based on a priority.\n+The compaction algorithm is continuously running and always selects files to compact based on a priority.\n \n 1. If there are closed WAL files, the 5 oldest WAL segments are added to the set of compaction files.\n-2. If any TSM files contain points with older timestamps that also exist in the WAL files, those TSM files are added to the compaction set.\n+2. If any TSM files contains points with older timestamps that also exist in the WAL files, those TSM files are added to the compaction set.","path":"tsdb/engine/tsm1/DESIGN.md","position":null,"original_position":53,"commit_id":"069944cf25d3eaf1f1c53dbead2802e039206341","original_commit_id":"936404e396351f407cd376fdfb146dc71ed2d7ba","user":{"login":"jonseymour","id":207131,"avatar_url":"https://avatars.githubusercontent.com/u/207131?v=3","gravatar_id":"","url":"https://api.github.com/users/jonseymour","html_url":"https://github.com/jonseymour","followers_url":"https://api.github.com/users/jonseymour/followers","following_url":"https://api.github.com/users/jonseymour/following{/other_user}","gists_url":"https://api.github.com/users/jonseymour/gists{/gist_id}","starred_url":"https://api.github.com/users/jonseymour/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonseymour/subscriptions","organizations_url":"https://api.github.com/users/jonseymour/orgs","repos_url":"https://api.github.com/users/jonseymour/repos","events_url":"https://api.github.com/users/jonseymour/events{/privacy}","received_events_url":"https://api.github.com/users/jonseymour/received_events","type":"User","site_admin":false},"body":"incorrect edit - undone with 069944c to be squashed later.","created_at":"2016-02-13T10:16:30Z","updated_at":"2016-02-13T10:16:30Z","html_url":"https://github.com/influxdata/influxdb/pull/5678#discussion_r52826089","pull_request_url":"https://api.github.com/repos/influxdata/influxdb/pulls/5678","_links":{"self":{"href":"https://api.github.com/repos/influxdata/influxdb/pulls/comments/52826089"},"html":{"href":"https://github.com/influxdata/influxdb/pull/5678#discussion_r52826089"},"pull_request":{"href":"https://api.github.com/repos/influxdata/influxdb/pulls/5678"}}},"pull_request":{"url":"https://api.github.com/repos/influxdata/influxdb/pulls/5678","id":59259208,"html_url":"https://github.com/influxdata/influxdb/pull/5678","diff_url":"https://github.com/influxdata/influxdb/pull/5678.diff","patch_url":"https://github.com/influxdata/influxdb/pull/5678.patch","issue_url":"https://api.github.com/repos/influxdata/influxdb/issues/5678","number":5678,"state":"open","locked":false,"title":"doc: typographical, spelling, grammar, word-choice and phrasing improvements.","user":{"login":"jonseymour","id":207131,"avatar_url":"https://avatars.githubusercontent.com/u/207131?v=3","gravatar_id":"","url":"https://api.github.com/users/jonseymour","html_url":"https://github.com/jonseymour","followers_url":"https://api.github.com/users/jonseymour/followers","following_url":"https://api.github.com/users/jonseymour/following{/other_user}","gists_url":"https://api.github.com/users/jonseymour/gists{/gist_id}","starred_url":"https://api.github.com/users/jonseymour/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonseymour/subscriptions","organizations_url":"https://api.github.com/users/jonseymour/orgs","repos_url":"https://api.github.com/users/jonseymour/repos","events_url":"https://api.github.com/users/jonseymour/events{/privacy}","received_events_url":"https://api.github.com/users/jonseymour/received_events","type":"User","site_admin":false},"body":"This change fixes various typographical, spelling, grammar and word-choice issues.\r\n\r\nThe earlier commits in the series are fairly mechanical spelling and typo and grammar corrections. The later commits are an attempt to make the text read more naturally.","created_at":"2016-02-13T05:21:34Z","updated_at":"2016-02-13T10:16:30Z","closed_at":null,"merged_at":null,"merge_commit_sha":"342181a4805e8c9f47b1182e4f975d3c910e8e6c","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/influxdata/influxdb/pulls/5678/commits","review_comments_url":"https://api.github.com/repos/influxdata/influxdb/pulls/5678/comments","review_comment_url":"https://api.github.com/repos/influxdata/influxdb/pulls/comments{/number}","comments_url":"https://api.github.com/repos/influxdata/influxdb/issues/5678/comments","statuses_url":"https://api.github.com/repos/influxdata/influxdb/statuses/069944cf25d3eaf1f1c53dbead2802e039206341","head":{"label":"jonseymour:typo","ref":"typo","sha":"069944cf25d3eaf1f1c53dbead2802e039206341","user":{"login":"jonseymour","id":207131,"avatar_url":"https://avatars.githubusercontent.com/u/207131?v=3","gravatar_id":"","url":"https://api.github.com/users/jonseymour","html_url":"https://github.com/jonseymour","followers_url":"https://api.github.com/users/jonseymour/followers","following_url":"https://api.github.com/users/jonseymour/following{/other_user}","gists_url":"https://api.github.com/users/jonseymour/gists{/gist_id}","starred_url":"https://api.github.com/users/jonseymour/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonseymour/subscriptions","organizations_url":"https://api.github.com/users/jonseymour/orgs","repos_url":"https://api.github.com/users/jonseymour/repos","events_url":"https://api.github.com/users/jonseymour/events{/privacy}","received_events_url":"https://api.github.com/users/jonseymour/received_events","type":"User","site_admin":false},"repo":{"id":40862560,"name":"influxdb","full_name":"jonseymour/influxdb","owner":{"login":"jonseymour","id":207131,"avatar_url":"https://avatars.githubusercontent.com/u/207131?v=3","gravatar_id":"","url":"https://api.github.com/users/jonseymour","html_url":"https://github.com/jonseymour","followers_url":"https://api.github.com/users/jonseymour/followers","following_url":"https://api.github.com/users/jonseymour/following{/other_user}","gists_url":"https://api.github.com/users/jonseymour/gists{/gist_id}","starred_url":"https://api.github.com/users/jonseymour/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonseymour/subscriptions","organizations_url":"https://api.github.com/users/jonseymour/orgs","repos_url":"https://api.github.com/users/jonseymour/repos","events_url":"https://api.github.com/users/jonseymour/events{/privacy}","received_events_url":"https://api.github.com/users/jonseymour/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jonseymour/influxdb","description":"Scalable datastore for metrics, events, and real-time analytics","fork":true,"url":"https://api.github.com/repos/jonseymour/influxdb","forks_url":"https://api.github.com/repos/jonseymour/influxdb/forks","keys_url":"https://api.github.com/repos/jonseymour/influxdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jonseymour/influxdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jonseymour/influxdb/teams","hooks_url":"https://api.github.com/repos/jonseymour/influxdb/hooks","issue_events_url":"https://api.github.com/repos/jonseymour/influxdb/issues/events{/number}","events_url":"https://api.github.com/repos/jonseymour/influxdb/events","assignees_url":"https://api.github.com/repos/jonseymour/influxdb/assignees{/user}","branches_url":"https://api.github.com/repos/jonseymour/influxdb/branches{/branch}","tags_url":"https://api.github.com/repos/jonseymour/influxdb/tags","blobs_url":"https://api.github.com/repos/jonseymour/influxdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jonseymour/influxdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jonseymour/influxdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/jonseymour/influxdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jonseymour/influxdb/statuses/{sha}","languages_url":"https://api.github.com/repos/jonseymour/influxdb/languages","stargazers_url":"https://api.github.com/repos/jonseymour/influxdb/stargazers","contributors_url":"https://api.github.com/repos/jonseymour/influxdb/contributors","subscribers_url":"https://api.github.com/repos/jonseymour/influxdb/subscribers","subscription_url":"https://api.github.com/repos/jonseymour/influxdb/subscription","commits_url":"https://api.github.com/repos/jonseymour/influxdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/jonseymour/influxdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/jonseymour/influxdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/jonseymour/influxdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/jonseymour/influxdb/contents/{+path}","compare_url":"https://api.github.com/repos/jonseymour/influxdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jonseymour/influxdb/merges","archive_url":"https://api.github.com/repos/jonseymour/influxdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jonseymour/influxdb/downloads","issues_url":"https://api.github.com/repos/jonseymour/influxdb/issues{/number}","pulls_url":"https://api.github.com/repos/jonseymour/influxdb/pulls{/number}","milestones_url":"https://api.github.com/repos/jonseymour/influxdb/milestones{/number}","notifications_url":"https://api.github.com/repos/jonseymour/influxdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jonseymour/influxdb/labels{/name}","releases_url":"https://api.github.com/repos/jonseymour/influxdb/releases{/id}","deployments_url":"https://api.github.com/repos/jonseymour/influxdb/deployments","created_at":"2015-08-17T04:08:54Z","updated_at":"2015-08-17T04:08:56Z","pushed_at":"2016-02-13T10:16:12Z","git_url":"git://github.com/jonseymour/influxdb.git","ssh_url":"git@github.com:jonseymour/influxdb.git","clone_url":"https://github.com/jonseymour/influxdb.git","svn_url":"https://github.com/jonseymour/influxdb","homepage":"http://influxdb.com","size":36139,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"influxdata:master","ref":"master","sha":"8e100998fab81a70f8b8fa8fbb41e1ed8cafa6f5","user":{"login":"influxdata","id":5713248,"avatar_url":"https://avatars.githubusercontent.com/u/5713248?v=3","gravatar_id":"","url":"https://api.github.com/users/influxdata","html_url":"https://github.com/influxdata","followers_url":"https://api.github.com/users/influxdata/followers","following_url":"https://api.github.com/users/influxdata/following{/other_user}","gists_url":"https://api.github.com/users/influxdata/gists{/gist_id}","starred_url":"https://api.github.com/users/influxdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/influxdata/subscriptions","organizations_url":"https://api.github.com/users/influxdata/orgs","repos_url":"https://api.github.com/users/influxdata/repos","events_url":"https://api.github.com/users/influxdata/events{/privacy}","received_events_url":"https://api.github.com/users/influxdata/received_events","type":"Organization","site_admin":false},"repo":{"id":13124802,"name":"influxdb","full_name":"influxdata/influxdb","owner":{"login":"influxdata","id":5713248,"avatar_url":"https://avatars.githubusercontent.com/u/5713248?v=3","gravatar_id":"","url":"https://api.github.com/users/influxdata","html_url":"https://github.com/influxdata","followers_url":"https://api.github.com/users/influxdata/followers","following_url":"https://api.github.com/users/influxdata/following{/other_user}","gists_url":"https://api.github.com/users/influxdata/gists{/gist_id}","starred_url":"https://api.github.com/users/influxdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/influxdata/subscriptions","organizations_url":"https://api.github.com/users/influxdata/orgs","repos_url":"https://api.github.com/users/influxdata/repos","events_url":"https://api.github.com/users/influxdata/events{/privacy}","received_events_url":"https://api.github.com/users/influxdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/influxdata/influxdb","description":"Scalable datastore for metrics, events, and real-time analytics","fork":false,"url":"https://api.github.com/repos/influxdata/influxdb","forks_url":"https://api.github.com/repos/influxdata/influxdb/forks","keys_url":"https://api.github.com/repos/influxdata/influxdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/influxdata/influxdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/influxdata/influxdb/teams","hooks_url":"https://api.github.com/repos/influxdata/influxdb/hooks","issue_events_url":"https://api.github.com/repos/influxdata/influxdb/issues/events{/number}","events_url":"https://api.github.com/repos/influxdata/influxdb/events","assignees_url":"https://api.github.com/repos/influxdata/influxdb/assignees{/user}","branches_url":"https://api.github.com/repos/influxdata/influxdb/branches{/branch}","tags_url":"https://api.github.com/repos/influxdata/influxdb/tags","blobs_url":"https://api.github.com/repos/influxdata/influxdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/influxdata/influxdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/influxdata/influxdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/influxdata/influxdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/influxdata/influxdb/statuses/{sha}","languages_url":"https://api.github.com/repos/influxdata/influxdb/languages","stargazers_url":"https://api.github.com/repos/influxdata/influxdb/stargazers","contributors_url":"https://api.github.com/repos/influxdata/influxdb/contributors","subscribers_url":"https://api.github.com/repos/influxdata/influxdb/subscribers","subscription_url":"https://api.github.com/repos/influxdata/influxdb/subscription","commits_url":"https://api.github.com/repos/influxdata/influxdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/influxdata/influxdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/influxdata/influxdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/influxdata/influxdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/influxdata/influxdb/contents/{+path}","compare_url":"https://api.github.com/repos/influxdata/influxdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/influxdata/influxdb/merges","archive_url":"https://api.github.com/repos/influxdata/influxdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/influxdata/influxdb/downloads","issues_url":"https://api.github.com/repos/influxdata/influxdb/issues{/number}","pulls_url":"https://api.github.com/repos/influxdata/influxdb/pulls{/number}","milestones_url":"https://api.github.com/repos/influxdata/influxdb/milestones{/number}","notifications_url":"https://api.github.com/repos/influxdata/influxdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/influxdata/influxdb/labels{/name}","releases_url":"https://api.github.com/repos/influxdata/influxdb/releases{/id}","deployments_url":"https://api.github.com/repos/influxdata/influxdb/deployments","created_at":"2013-09-26T14:31:10Z","updated_at":"2016-02-13T05:40:49Z","pushed_at":"2016-02-13T10:16:12Z","git_url":"git://github.com/influxdata/influxdb.git","ssh_url":"git@github.com:influxdata/influxdb.git","clone_url":"https://github.com/influxdata/influxdb.git","svn_url":"https://github.com/influxdata/influxdb","homepage":"https://influxdata.com","size":41465,"stargazers_count":7362,"watchers_count":7362,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":936,"mirror_url":null,"open_issues_count":590,"forks":936,"open_issues":590,"watchers":7362,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/influxdata/influxdb/pulls/5678"},"html":{"href":"https://github.com/influxdata/influxdb/pull/5678"},"issue":{"href":"https://api.github.com/repos/influxdata/influxdb/issues/5678"},"comments":{"href":"https://api.github.com/repos/influxdata/influxdb/issues/5678/comments"},"review_comments":{"href":"https://api.github.com/repos/influxdata/influxdb/pulls/5678/comments"},"review_comment":{"href":"https://api.github.com/repos/influxdata/influxdb/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/influxdata/influxdb/pulls/5678/commits"},"statuses":{"href":"https://api.github.com/repos/influxdata/influxdb/statuses/069944cf25d3eaf1f1c53dbead2802e039206341"}}}}