{"action":"created","comment":{"url":"https://api.github.com/repos/apache/flink/pulls/comments/69722894","id":69722894,"diff_hunk":"@@ -0,0 +1,135 @@\n+/*\n+ * Licensed to the Apache Software Foundation (ASF) under one\n+ * or more contributor license agreements.  See the NOTICE file\n+ * distributed with this work for additional information\n+ * regarding copyright ownership.  The ASF licenses this file\n+ * to you under the Apache License, Version 2.0 (the\n+ * \"License\"); you may not use this file except in compliance\n+ * with the License.  You may obtain a copy of the License at\n+ *\n+ *     http://www.apache.org/licenses/LICENSE-2.0\n+ *\n+ * Unless required by applicable law or agreed to in writing, software\n+ * distributed under the License is distributed on an \"AS IS\" BASIS,\n+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n+ * See the License for the specific language governing permissions and\n+ * limitations under the License.\n+ */\n+\n+package org.apache.flink.runtime.checkpoint;\n+\n+import org.apache.flink.api.common.JobID;\n+import org.apache.flink.runtime.jobgraph.JobVertexID;\n+import org.apache.flink.util.Preconditions;\n+\n+import java.io.Serializable;\n+import java.util.Map;\n+import java.util.Objects;\n+\n+/**\n+ * A successful checkpoint describes a checkpoint after all required tasks acknowledged it (with their state)\n+ * and that is considered completed.\n+ */\n+public class Checkpoint implements Serializable {\n+\n+\tprivate static final long serialVersionUID = -8360248179615702014L;\n+\n+\tprivate final JobID job;\n+\t\n+\tprivate final long checkpointID;\n+\n+\t/** The timestamp when the checkpoint was triggered. */\n+\tprivate final long timestamp;\n+\n+\t/** The duration of the checkpoint (completion timestamp - trigger timestamp). */\n+\tprivate final long duration;\n+\n+\t/** States of the different task groups belonging to this checkpoint */\n+\tprivate final Map<JobVertexID, TaskState> taskStates;\n+\n+\tpublic Checkpoint(\n+\t\tJobID job,\n+\t\tlong checkpointID,\n+\t\tlong timestamp,\n+\t\tlong completionTimestamp,\n+\t\tMap<JobVertexID, TaskState> taskStates) {\n+\n+\t\tthis.job = job;","path":"flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/Checkpoint.java","position":57,"original_position":57,"commit_id":"af0d418ba2bc3591f0526476d1ad7dfc4160c205","original_commit_id":"af0d418ba2bc3591f0526476d1ad7dfc4160c205","user":{"login":"uce","id":1756620,"avatar_url":"https://avatars.githubusercontent.com/u/1756620?v=3","gravatar_id":"","url":"https://api.github.com/users/uce","html_url":"https://github.com/uce","followers_url":"https://api.github.com/users/uce/followers","following_url":"https://api.github.com/users/uce/following{/other_user}","gists_url":"https://api.github.com/users/uce/gists{/gist_id}","starred_url":"https://api.github.com/users/uce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uce/subscriptions","organizations_url":"https://api.github.com/users/uce/orgs","repos_url":"https://api.github.com/users/uce/repos","events_url":"https://api.github.com/users/uce/events{/privacy}","received_events_url":"https://api.github.com/users/uce/received_events","type":"User","site_admin":false},"body":"Will do (this is an unrelated change though as `Checkpoint` is a renaming of the current `master`'s `CompletedCheckpoint`)","created_at":"2016-07-06T12:45:51Z","updated_at":"2016-07-06T12:45:51Z","html_url":"https://github.com/apache/flink/pull/2194#discussion_r69722894","pull_request_url":"https://api.github.com/repos/apache/flink/pulls/2194","_links":{"self":{"href":"https://api.github.com/repos/apache/flink/pulls/comments/69722894"},"html":{"href":"https://github.com/apache/flink/pull/2194#discussion_r69722894"},"pull_request":{"href":"https://api.github.com/repos/apache/flink/pulls/2194"}}},"pull_request":{"url":"https://api.github.com/repos/apache/flink/pulls/2194","id":76093679,"html_url":"https://github.com/apache/flink/pull/2194","diff_url":"https://github.com/apache/flink/pull/2194.diff","patch_url":"https://github.com/apache/flink/pull/2194.patch","issue_url":"https://api.github.com/repos/apache/flink/issues/2194","number":2194,"state":"open","locked":false,"title":"[FLINK-4067] [runtime] Add savepoint headers","user":{"login":"uce","id":1756620,"avatar_url":"https://avatars.githubusercontent.com/u/1756620?v=3","gravatar_id":"","url":"https://api.github.com/users/uce","html_url":"https://github.com/uce","followers_url":"https://api.github.com/users/uce/followers","following_url":"https://api.github.com/users/uce/following{/other_user}","gists_url":"https://api.github.com/users/uce/gists{/gist_id}","starred_url":"https://api.github.com/users/uce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uce/subscriptions","organizations_url":"https://api.github.com/users/uce/orgs","repos_url":"https://api.github.com/users/uce/repos","events_url":"https://api.github.com/users/uce/events{/privacy}","received_events_url":"https://api.github.com/users/uce/received_events","type":"User","site_admin":false},"body":"Savepoints were previously persisted without any meta data using default Java serialization of a `CompletedCheckpoint`. This PR introduces a savepoint class with version-specific serializers and stores savepoints with meta data.\r\n\r\nSavepoints expose a version number and a `Collection<TaskState>` for savepoint restore.\r\n\r\nCurrently, there are two savepoint versions:\r\n\r\n- `SavepointV0` (Flink 1.0): This is a wrapper around the Flink 1.0 `CompletedCheckpoint`. Because this class changed since Flink 1.0 I had to re-introduce it with this PR in order to allow restoring from Flink 1.0 savepoints (otherwise the old `StateForTask` state can not be deserialized via default Java serialization). Therefore, I've renamed the Flink 1.1 `CompletedCheckpoint` to `Checkpoint`. I've annotated the re-introduced classes with `@Deprecated`. We can remove those with Flink 1.2.\r\n- `SavepointV1` (Flink 1.1): This is the current savepoint version, which holds a reference to the `Checkpoint` task state collection, but is serialized with a custom serializater not relying on default Java serialization. Therefore, it should not happen again that we need to stick to certain classes in future Flink versions.\r\n\r\nThe savepoints are stored in `FsSavepointStore` with the following format:\r\n\r\n```\r\nMagicNumber SavepointVersion Savepoint\r\n  - MagicNumber => int\r\n  - SavepointVersion => int (returned by Savepoint#getVersion())\r\n  - Savepoint => bytes (serialized via version-specific SavepointSerializer)\r\n```\r\n\r\nThe header is minimal (magic number, version). All savepoint-specific meta data can be moved to the savepoint itself. This is also were we would have to add new meta data in future versions, allowing us to differentiate between different savepoint versions when we change the serialization stack etc.\r\n\r\nI've tested both manually and via a unit test that it works to trigger a savepoint via Flink 1.0 and resume from it with Flink 1.1.\r\n\r\nAll savepoint related classes have been moved from `checkpoint` to a new sub package `checkpoint.savepoint`.\r\n\r\nThe main classes to look at to review this PR are `Savepoint` and subclasses, `SavepointSerializer` and subclasses, and `FsSavepointStore`. @tillrohrmann do you have time to do this?","created_at":"2016-07-03T17:08:32Z","updated_at":"2016-07-06T12:45:51Z","closed_at":null,"merged_at":null,"merge_commit_sha":"00313be0fc63e257682b619462f194337a37af18","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/flink/pulls/2194/commits","review_comments_url":"https://api.github.com/repos/apache/flink/pulls/2194/comments","review_comment_url":"https://api.github.com/repos/apache/flink/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/flink/issues/2194/comments","statuses_url":"https://api.github.com/repos/apache/flink/statuses/af0d418ba2bc3591f0526476d1ad7dfc4160c205","head":{"label":"uce:4067-savepoint_header","ref":"4067-savepoint_header","sha":"af0d418ba2bc3591f0526476d1ad7dfc4160c205","user":{"login":"uce","id":1756620,"avatar_url":"https://avatars.githubusercontent.com/u/1756620?v=3","gravatar_id":"","url":"https://api.github.com/users/uce","html_url":"https://github.com/uce","followers_url":"https://api.github.com/users/uce/followers","following_url":"https://api.github.com/users/uce/following{/other_user}","gists_url":"https://api.github.com/users/uce/gists{/gist_id}","starred_url":"https://api.github.com/users/uce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uce/subscriptions","organizations_url":"https://api.github.com/users/uce/orgs","repos_url":"https://api.github.com/users/uce/repos","events_url":"https://api.github.com/users/uce/events{/privacy}","received_events_url":"https://api.github.com/users/uce/received_events","type":"User","site_admin":false},"repo":{"id":20699435,"name":"flink","full_name":"uce/flink","owner":{"login":"uce","id":1756620,"avatar_url":"https://avatars.githubusercontent.com/u/1756620?v=3","gravatar_id":"","url":"https://api.github.com/users/uce","html_url":"https://github.com/uce","followers_url":"https://api.github.com/users/uce/followers","following_url":"https://api.github.com/users/uce/following{/other_user}","gists_url":"https://api.github.com/users/uce/gists{/gist_id}","starred_url":"https://api.github.com/users/uce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uce/subscriptions","organizations_url":"https://api.github.com/users/uce/orgs","repos_url":"https://api.github.com/users/uce/repos","events_url":"https://api.github.com/users/uce/events{/privacy}","received_events_url":"https://api.github.com/users/uce/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/uce/flink","description":"Mirror of Apache Flink","fork":true,"url":"https://api.github.com/repos/uce/flink","forks_url":"https://api.github.com/repos/uce/flink/forks","keys_url":"https://api.github.com/repos/uce/flink/keys{/key_id}","collaborators_url":"https://api.github.com/repos/uce/flink/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/uce/flink/teams","hooks_url":"https://api.github.com/repos/uce/flink/hooks","issue_events_url":"https://api.github.com/repos/uce/flink/issues/events{/number}","events_url":"https://api.github.com/repos/uce/flink/events","assignees_url":"https://api.github.com/repos/uce/flink/assignees{/user}","branches_url":"https://api.github.com/repos/uce/flink/branches{/branch}","tags_url":"https://api.github.com/repos/uce/flink/tags","blobs_url":"https://api.github.com/repos/uce/flink/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/uce/flink/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/uce/flink/git/refs{/sha}","trees_url":"https://api.github.com/repos/uce/flink/git/trees{/sha}","statuses_url":"https://api.github.com/repos/uce/flink/statuses/{sha}","languages_url":"https://api.github.com/repos/uce/flink/languages","stargazers_url":"https://api.github.com/repos/uce/flink/stargazers","contributors_url":"https://api.github.com/repos/uce/flink/contributors","subscribers_url":"https://api.github.com/repos/uce/flink/subscribers","subscription_url":"https://api.github.com/repos/uce/flink/subscription","commits_url":"https://api.github.com/repos/uce/flink/commits{/sha}","git_commits_url":"https://api.github.com/repos/uce/flink/git/commits{/sha}","comments_url":"https://api.github.com/repos/uce/flink/comments{/number}","issue_comment_url":"https://api.github.com/repos/uce/flink/issues/comments{/number}","contents_url":"https://api.github.com/repos/uce/flink/contents/{+path}","compare_url":"https://api.github.com/repos/uce/flink/compare/{base}...{head}","merges_url":"https://api.github.com/repos/uce/flink/merges","archive_url":"https://api.github.com/repos/uce/flink/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/uce/flink/downloads","issues_url":"https://api.github.com/repos/uce/flink/issues{/number}","pulls_url":"https://api.github.com/repos/uce/flink/pulls{/number}","milestones_url":"https://api.github.com/repos/uce/flink/milestones{/number}","notifications_url":"https://api.github.com/repos/uce/flink/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/uce/flink/labels{/name}","releases_url":"https://api.github.com/repos/uce/flink/releases{/id}","deployments_url":"https://api.github.com/repos/uce/flink/deployments","created_at":"2014-06-10T19:56:02Z","updated_at":"2016-05-31T08:52:12Z","pushed_at":"2016-07-06T12:33:17Z","git_url":"git://github.com/uce/flink.git","ssh_url":"git@github.com:uce/flink.git","clone_url":"https://github.com/uce/flink.git","svn_url":"https://github.com/uce/flink","homepage":null,"size":90152,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"bdfcf10c4764b44575844730fc9d900ec42a8dcf","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":20587599,"name":"flink","full_name":"apache/flink","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/flink","description":"Mirror of Apache Flink","fork":false,"url":"https://api.github.com/repos/apache/flink","forks_url":"https://api.github.com/repos/apache/flink/forks","keys_url":"https://api.github.com/repos/apache/flink/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/flink/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/flink/teams","hooks_url":"https://api.github.com/repos/apache/flink/hooks","issue_events_url":"https://api.github.com/repos/apache/flink/issues/events{/number}","events_url":"https://api.github.com/repos/apache/flink/events","assignees_url":"https://api.github.com/repos/apache/flink/assignees{/user}","branches_url":"https://api.github.com/repos/apache/flink/branches{/branch}","tags_url":"https://api.github.com/repos/apache/flink/tags","blobs_url":"https://api.github.com/repos/apache/flink/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/flink/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/flink/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/flink/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/flink/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/flink/languages","stargazers_url":"https://api.github.com/repos/apache/flink/stargazers","contributors_url":"https://api.github.com/repos/apache/flink/contributors","subscribers_url":"https://api.github.com/repos/apache/flink/subscribers","subscription_url":"https://api.github.com/repos/apache/flink/subscription","commits_url":"https://api.github.com/repos/apache/flink/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/flink/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/flink/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/flink/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/flink/contents/{+path}","compare_url":"https://api.github.com/repos/apache/flink/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/flink/merges","archive_url":"https://api.github.com/repos/apache/flink/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/flink/downloads","issues_url":"https://api.github.com/repos/apache/flink/issues{/number}","pulls_url":"https://api.github.com/repos/apache/flink/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/flink/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/flink/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/flink/labels{/name}","releases_url":"https://api.github.com/repos/apache/flink/releases{/id}","deployments_url":"https://api.github.com/repos/apache/flink/deployments","created_at":"2014-06-07T07:00:10Z","updated_at":"2016-07-06T06:10:56Z","pushed_at":"2016-07-06T12:43:04Z","git_url":"git://github.com/apache/flink.git","ssh_url":"git@github.com:apache/flink.git","clone_url":"https://github.com/apache/flink.git","svn_url":"https://github.com/apache/flink","homepage":null,"size":121440,"stargazers_count":1408,"watchers_count":1408,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":914,"mirror_url":"git://git.apache.org/flink.git","open_issues_count":110,"forks":914,"open_issues":110,"watchers":1408,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/flink/pulls/2194"},"html":{"href":"https://github.com/apache/flink/pull/2194"},"issue":{"href":"https://api.github.com/repos/apache/flink/issues/2194"},"comments":{"href":"https://api.github.com/repos/apache/flink/issues/2194/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/flink/pulls/2194/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/flink/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/flink/pulls/2194/commits"},"statuses":{"href":"https://api.github.com/repos/apache/flink/statuses/af0d418ba2bc3591f0526476d1ad7dfc4160c205"}}}}
{"actor":{"display_login":"uce"}}