{"action":"created","issue":{"url":"https://api.github.com/repos/apache/flink/issues/6302","repository_url":"https://api.github.com/repos/apache/flink","labels_url":"https://api.github.com/repos/apache/flink/issues/6302/labels{/name}","comments_url":"https://api.github.com/repos/apache/flink/issues/6302/comments","events_url":"https://api.github.com/repos/apache/flink/issues/6302/events","html_url":"https://github.com/apache/flink/pull/6302","id":340082228,"node_id":"MDExOlB1bGxSZXF1ZXN0MjAwNTc5OTE5","number":6302,"title":" [FLINK-9061][checkpointing] add entropy to s3 path for better scalability","user":{"login":"indrc","id":25802702,"node_id":"MDQ6VXNlcjI1ODAyNzAy","avatar_url":"https://avatars2.githubusercontent.com/u/25802702?v=4","gravatar_id":"","url":"https://api.github.com/users/indrc","html_url":"https://github.com/indrc","followers_url":"https://api.github.com/users/indrc/followers","following_url":"https://api.github.com/users/indrc/following{/other_user}","gists_url":"https://api.github.com/users/indrc/gists{/gist_id}","starred_url":"https://api.github.com/users/indrc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/indrc/subscriptions","organizations_url":"https://api.github.com/users/indrc/orgs","repos_url":"https://api.github.com/users/indrc/repos","events_url":"https://api.github.com/users/indrc/events{/privacy}","received_events_url":"https://api.github.com/users/indrc/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-11T03:51:46Z","updated_at":"2018-07-15T17:36:36Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/apache/flink/pulls/6302","html_url":"https://github.com/apache/flink/pull/6302","diff_url":"https://github.com/apache/flink/pull/6302.diff","patch_url":"https://github.com/apache/flink/pull/6302.patch"},"body":"## What is the purpose of the change\r\n\r\n*This pull request adds entropy to the checkpoint path based upon a user defined pattern in the configuration for better s3 scalability*\r\n\r\n\r\n## Brief change log\r\n  - *Read the entropy pattern from the config*\r\n  - *In the checkpoint path for FSbackend, substitute the entropy key with a 4 character random alpha numeric string*\r\n\r\n\r\n## Verifying this change\r\n\r\nThis change added tests and can be verified as follows:\r\n\r\n  - *Added unit tests under FileStateBackendTest to cover different scenarios for the entropy substitution*\r\n\r\n## Does this pull request potentially affect one of the following parts:\r\n\r\n  - Dependencies (does it add or upgrade a dependency): (yes )\r\n  - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no)\r\n  - The serializers: ( no )\r\n  - The runtime per-record code paths (performance sensitive): (no)\r\n  - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes )\r\n  - The S3 file system connector: (no)\r\n\r\n## Documentation\r\n\r\n  - Does this pull request introduce a new feature? (no)"},"comment":{"url":"https://api.github.com/repos/apache/flink/issues/comments/405106084","html_url":"https://github.com/apache/flink/pull/6302#issuecomment-405106084","issue_url":"https://api.github.com/repos/apache/flink/issues/6302","id":405106084,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNTEwNjA4NA==","user":{"login":"StephanEwen","id":1727146,"node_id":"MDQ6VXNlcjE3MjcxNDY=","avatar_url":"https://avatars3.githubusercontent.com/u/1727146?v=4","gravatar_id":"","url":"https://api.github.com/users/StephanEwen","html_url":"https://github.com/StephanEwen","followers_url":"https://api.github.com/users/StephanEwen/followers","following_url":"https://api.github.com/users/StephanEwen/following{/other_user}","gists_url":"https://api.github.com/users/StephanEwen/gists{/gist_id}","starred_url":"https://api.github.com/users/StephanEwen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StephanEwen/subscriptions","organizations_url":"https://api.github.com/users/StephanEwen/orgs","repos_url":"https://api.github.com/users/StephanEwen/repos","events_url":"https://api.github.com/users/StephanEwen/events{/privacy}","received_events_url":"https://api.github.com/users/StephanEwen/received_events","type":"User","site_admin":false},"created_at":"2018-07-15T17:36:36Z","updated_at":"2018-07-15T17:36:36Z","author_association":"CONTRIBUTOR","body":"Thanks for this contribution, that's a valuable fix. I have a few thoughts and suggestions on how we might improve the feature a bit still:\r\n\r\n  - Can we get id of the `commons-text` dependency? The fewer dependencies, the fewer possible problems for users due to dependency clashes. It seems a bit heavy to add a new library for just one random string generation.\r\n\r\n  - The feature is configured through additional constructor parameters. I am wondering if we may want to move this to the `Configuration`. That would allow the \"ops side of things\" to configure this for a setup (setting entropy key and checkpoints directory) without needing everyone that writes a Flink program to be aware of this.\r\n\r\n  - If I read the code correctly, the code logs warnings for every file in case the feature is not activated. That will probably confuse a lot of users and make them dig into whether they have a wrong setup, when they simply don't use this new feature.\r\n"}}
{"actor":{"display_login":"StephanEwen"}}