{"action":"created","issue":{"url":"https://api.github.com/repos/pravega/pravega/issues/2967","repository_url":"https://api.github.com/repos/pravega/pravega","labels_url":"https://api.github.com/repos/pravega/pravega/issues/2967/labels{/name}","comments_url":"https://api.github.com/repos/pravega/pravega/issues/2967/comments","events_url":"https://api.github.com/repos/pravega/pravega/issues/2967/events","html_url":"https://github.com/pravega/pravega/issues/2967","id":367803782,"node_id":"MDU6SXNzdWUzNjc4MDM3ODI=","number":2967,"title":"HDFS append errors observed in segment stores ","user":{"login":"kmgowda","id":7926048,"node_id":"MDQ6VXNlcjc5MjYwNDg=","avatar_url":"https://avatars0.githubusercontent.com/u/7926048?v=4","gravatar_id":"","url":"https://api.github.com/users/kmgowda","html_url":"https://github.com/kmgowda","followers_url":"https://api.github.com/users/kmgowda/followers","following_url":"https://api.github.com/users/kmgowda/following{/other_user}","gists_url":"https://api.github.com/users/kmgowda/gists{/gist_id}","starred_url":"https://api.github.com/users/kmgowda/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmgowda/subscriptions","organizations_url":"https://api.github.com/users/kmgowda/orgs","repos_url":"https://api.github.com/users/kmgowda/repos","events_url":"https://api.github.com/users/kmgowda/events{/privacy}","received_events_url":"https://api.github.com/users/kmgowda/received_events","type":"User","site_admin":false},"labels":[{"id":406474443,"node_id":"MDU6TGFiZWw0MDY0NzQ0NDM=","url":"https://api.github.com/repos/pravega/pravega/labels/area/tier2","name":"area/tier2","color":"c2e0c6","default":false},{"id":406474445,"node_id":"MDU6TGFiZWw0MDY0NzQ0NDU=","url":"https://api.github.com/repos/pravega/pravega/labels/kind/question","name":"kind/question","color":"fef2c0","default":false},{"id":520512698,"node_id":"MDU6TGFiZWw1MjA1MTI2OTg=","url":"https://api.github.com/repos/pravega/pravega/labels/priority/P2","name":"priority/P2","color":"f9d0c4","default":false},{"id":515335315,"node_id":"MDU6TGFiZWw1MTUzMzUzMTU=","url":"https://api.github.com/repos/pravega/pravega/labels/status/needs-attention","name":"status/needs-attention","color":"FFFFE0","default":false}],"state":"open","locked":false,"assignee":{"login":"arvindkandhare","id":2305381,"node_id":"MDQ6VXNlcjIzMDUzODE=","avatar_url":"https://avatars1.githubusercontent.com/u/2305381?v=4","gravatar_id":"","url":"https://api.github.com/users/arvindkandhare","html_url":"https://github.com/arvindkandhare","followers_url":"https://api.github.com/users/arvindkandhare/followers","following_url":"https://api.github.com/users/arvindkandhare/following{/other_user}","gists_url":"https://api.github.com/users/arvindkandhare/gists{/gist_id}","starred_url":"https://api.github.com/users/arvindkandhare/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arvindkandhare/subscriptions","organizations_url":"https://api.github.com/users/arvindkandhare/orgs","repos_url":"https://api.github.com/users/arvindkandhare/repos","events_url":"https://api.github.com/users/arvindkandhare/events{/privacy}","received_events_url":"https://api.github.com/users/arvindkandhare/received_events","type":"User","site_admin":false},"assignees":[{"login":"arvindkandhare","id":2305381,"node_id":"MDQ6VXNlcjIzMDUzODE=","avatar_url":"https://avatars1.githubusercontent.com/u/2305381?v=4","gravatar_id":"","url":"https://api.github.com/users/arvindkandhare","html_url":"https://github.com/arvindkandhare","followers_url":"https://api.github.com/users/arvindkandhare/followers","following_url":"https://api.github.com/users/arvindkandhare/following{/other_user}","gists_url":"https://api.github.com/users/arvindkandhare/gists{/gist_id}","starred_url":"https://api.github.com/users/arvindkandhare/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arvindkandhare/subscriptions","organizations_url":"https://api.github.com/users/arvindkandhare/orgs","repos_url":"https://api.github.com/users/arvindkandhare/repos","events_url":"https://api.github.com/users/arvindkandhare/events{/privacy}","received_events_url":"https://api.github.com/users/arvindkandhare/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-10-08T13:58:43Z","updated_at":"2018-10-29T14:20:00Z","closed_at":null,"author_association":"NONE","body":"**Problem description**\r\nEventhough Pravega is funtional/operating.. \r\nduring segment writes, the segment store is throwing following errors:\r\n\r\n```\r\n2018-10-08 08:43:29,099 540577 [storage-io-16] ERROR i.p.s.server.writer.StorageWriter - StorageWriter[8]: Iteration[361].Error.\r\norg.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: Failed to APPEND_FILE /Scope/StreamName-4/0.#epoch.0$attributes$offset.0_2 for DFSClient_NONMAPREDUCE_-293683934_124 on 10.240.135.50 because DFSClient_NONMAPREDUCE_-293683934_124 is already the current lease holder.\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:3140)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:2935)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:3247)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:3216)\r\n        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:767)\r\n        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:433)\r\n        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\r\n        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)\r\n        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)\r\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)\r\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)\r\n        at java.security.AccessController.doPrivileged(Native Method)\r\n        at javax.security.auth.Subject.doAs(Subject.java:422)\r\n        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)\r\n        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)\r\n        at sun.reflect.GeneratedConstructorAccessor30.newInstance(Unknown Source)\r\n        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\r\n        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\r\n        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)\r\n        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)\r\n        at io.pravega.segmentstore.storage.impl.hdfs.HDFSExceptionHelpers.convertException(HDFSExceptionHelpers.java:37)\r\n        at io.pravega.segmentstore.storage.impl.hdfs.HDFSStorage.write(HDFSStorage.java:382)\r\n        at io.pravega.segmentstore.storage.rolling.RollingStorage.write(RollingStorage.java:322)\r\n        at io.pravega.segmentstore.storage.AsyncStorageWrapper.lambda$write$2(AsyncStorageWrapper.java:100)\r\n        at io.pravega.segmentstore.storage.AsyncStorageWrapper.lambda$runAsync$15(AsyncStorageWrapper.java:206)\r\n        at io.pravega.segmentstore.storage.AsyncStorageWrapper.execute(AsyncStorageWrapper.java:221)\r\n        at io.pravega.segmentstore.storage.AsyncStorageWrapper.lambda$supplyAsync$13(AsyncStorageWrapper.java:181)\r\n        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)\r\n        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\r\n        at java.util.concurrent.FutureTask.run(FutureTask.java:266)\r\n        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)\r\n        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n        at java.lang.Thread.run(Thread.java:748)\r\nCaused by: org.apache.hadoop.ipc.RemoteException: Failed to APPEND_FILE /Scope/StreamName-4/0.#epoch.0$attributes$offset.0_2 for DFSClient_NONMAPREDUCE_-293683934_124 on 10.240.135.50 because DFSClient_NONMAPREDUCE_-293683934_124 is already the current lease holder.\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:3140)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:2935)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:3247)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:3216)\r\n        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:767)\r\n        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:433)\r\n        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\r\n        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)\r\n        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)\r\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)\r\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)\r\n        at java.security.AccessController.doPrivileged(Native Method)\r\n        at javax.security.auth.Subject.doAs(Subject.java:422)\r\n        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)\r\n        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)\r\n        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481)\r\n        at org.apache.hadoop.ipc.Client.call(Client.java:1427)\r\n        at org.apache.hadoop.ipc.Client.call(Client.java:1337)\r\n        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)\r\n        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)\r\n        at com.sun.proxy.$Proxy13.append(Unknown Source)\r\n        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.append(ClientNamenodeProtocolTranslatorPB.java:324)\r\n        at sun.reflect.GeneratedMethodAccessor48.invoke(Unknown Source)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:498)\r\n        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)\r\n        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)\r\n        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)\r\n        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)\r\n        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)\r\n        at com.sun.proxy.$Proxy14.append(Unknown Source)\r\n        at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:1366)\r\n        at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:1388)\r\n        at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1457)\r\n        at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1427)\r\n        at org.apache.hadoop.hdfs.DistributedFileSystem$5.doCall(DistributedFileSystem.java:362)\r\n        at org.apache.hadoop.hdfs.DistributedFileSystem$5.doCall(DistributedFileSystem.java:358)\r\n        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)\r\n        at org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:370)\r\n        at org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:339)\r\n        at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1177)\r\n        at io.pravega.segmentstore.storage.impl.hdfs.HDFSStorage.write(HDFSStorage.java:356)\r\n        ... 13 common frames omitted\r\n\r\n```\r\n\r\n\r\n**Problem location**\r\nsegment store\r\n\r\nThis an issue HDFS Write and its already reported here : \r\nhttps://issues.apache.org/jira/browse/HDFS-11367\r\n\r\n\r\n**Suggestions for an improvement**\r\n\r\nSo, this link : https://issues.apache.org/jira/browse/HDFS-11367\r\n\r\nsuggest that to use\r\n\r\n                   writer.close();\r\n                   writerMap.remove(outputFile.toUri().getPath());\r\n\r\nin the segment store  appropriately to close the writer stream\r\n\r\nthis should examined \r\n\r\nresolving this issue may improve the performance too.\r\n"},"comment":{"url":"https://api.github.com/repos/pravega/pravega/issues/comments/433927598","html_url":"https://github.com/pravega/pravega/issues/2967#issuecomment-433927598","issue_url":"https://api.github.com/repos/pravega/pravega/issues/2967","id":433927598,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMzkyNzU5OA==","user":{"login":"RaulGracia","id":717112,"node_id":"MDQ6VXNlcjcxNzExMg==","avatar_url":"https://avatars1.githubusercontent.com/u/717112?v=4","gravatar_id":"","url":"https://api.github.com/users/RaulGracia","html_url":"https://github.com/RaulGracia","followers_url":"https://api.github.com/users/RaulGracia/followers","following_url":"https://api.github.com/users/RaulGracia/following{/other_user}","gists_url":"https://api.github.com/users/RaulGracia/gists{/gist_id}","starred_url":"https://api.github.com/users/RaulGracia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RaulGracia/subscriptions","organizations_url":"https://api.github.com/users/RaulGracia/orgs","repos_url":"https://api.github.com/users/RaulGracia/repos","events_url":"https://api.github.com/users/RaulGracia/events{/privacy}","received_events_url":"https://api.github.com/users/RaulGracia/received_events","type":"User","site_admin":false},"created_at":"2018-10-29T14:20:00Z","updated_at":"2018-10-29T14:20:00Z","author_association":"CONTRIBUTOR","body":"@kmgowda could you perform the same test with a newer version of Pravega? In PR #2988 we added the handling of `AlreadyBeingCreatedException` similarly to `StreamSegmentExistsException`. Let's see if this kind of HDFS exceptions are minimized. "}}
{"actor":{"display_login":"RaulGracia"}}