{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/12435","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/12435/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/12435/comments","events_url":"https://api.github.com/repos/apache/spark/issues/12435/events","html_url":"https://github.com/apache/spark/pull/12435","id":148791237,"number":12435,"title":"[SPARK-14678][SQL]Add a file sink log to support versioning and compaction","user":{"login":"zsxwing","id":1000778,"avatar_url":"https://avatars.githubusercontent.com/u/1000778?v=3","gravatar_id":"","url":"https://api.github.com/users/zsxwing","html_url":"https://github.com/zsxwing","followers_url":"https://api.github.com/users/zsxwing/followers","following_url":"https://api.github.com/users/zsxwing/following{/other_user}","gists_url":"https://api.github.com/users/zsxwing/gists{/gist_id}","starred_url":"https://api.github.com/users/zsxwing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zsxwing/subscriptions","organizations_url":"https://api.github.com/users/zsxwing/orgs","repos_url":"https://api.github.com/users/zsxwing/repos","events_url":"https://api.github.com/users/zsxwing/events{/privacy}","received_events_url":"https://api.github.com/users/zsxwing/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2016-04-15T23:24:17Z","updated_at":"2016-04-19T19:39:40Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12435","html_url":"https://github.com/apache/spark/pull/12435","diff_url":"https://github.com/apache/spark/pull/12435.diff","patch_url":"https://github.com/apache/spark/pull/12435.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR adds a special log for FileStreamSink for two purposes:\r\n\r\n- Versioning. A future Spark version should be able to read the metadata of an old FileStreamSink.\r\n- Compaction. As reading from many small files is usually pretty slow, we should compact small metadata files into big files.\r\n\r\nFileStreamSinkLog has a new log format instead of Java serialization format. It will write one log file for each batch. The first line of the log file is the version number, and there are multiple JSON lines following. Each JSON line is a JSON format of FileLog.\r\n\r\nFileStreamSinkLog will compact log files every \"spark.sql.sink.file.log.compactLen\" batches into a big file. When doing a compact, it will read all history logs and merge them with the new batch. During the compaction, it will also delete the files that are deleted (marked by FileLog.action). When the reader uses allLogs to list all files, this method only returns the visible files (drops the deleted files).\r\n\r\n## How was this patch tested?\r\n\r\nFileStreamSinkLogSuite\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/212085912","html_url":"https://github.com/apache/spark/pull/12435#issuecomment-212085912","issue_url":"https://api.github.com/repos/apache/spark/issues/12435","id":212085912,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-04-19T19:39:40Z","updated_at":"2016-04-19T19:39:40Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56242/\nTest PASSed."}}