hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Date Fri, 18 May 2012 22:15:10 GMT

    [ https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279255#comment-13279255

Todd Lipcon commented on HDFS-744:

Quickly looked at the patch. A few notes:

- I don't think it's a good idea to change SequenceFile.syncFs() to actually call through
to hsync(), given how much more expensive it is. This would be a "performance-incompatible
change". As much as it sucks, maybe we need to deprecate syncFs() and add a new method, instead
-- eg one that takes a boolean or an enum explaining what level of sync is needed.
- For the javadoc on SYNC_BLOCK, should explain that, though it's similar to O_SYNC, it differs
in that it's only on close that it is synced, and recommend that the user call hsync() explicitly
after each write if true synchronous behavior is required
- TestHSync is missing license header
- Metrics: we should have some metrics on number of hsyncs performed at each DN, as well as
the time spent in FileChannel.force. The functional tests could then verify these metrics
are actually incremented after the hsync calls where expected
- It seems wrong that the syncBlock flag is on the packets themselves. Why do we need this
flag? Why not just have the client (or server) keep a flag which gets set whenever hsync()
has been called. Then, when the client sends "last packet in block", the sync flag also gets
- Can you explain this logic to me?
+      if (syncPacket && !(syncBlock && lastPacketInBlock)) {
+        flushOrSync(true);
+      }
Is that to avoid the cost of a "double" sync on close? If so, can you add a comment as much?

+  required bool syncBlock = 5;
+  required bool syncPacket = 6;
These flags should be optional with a default of false, so that we don't break client-server
compatibility with 2.0.0-alpha.
> Support hsync in HDFS
>                 Key: HDFS-744
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client
>            Reporter: Hairong Kuang
>            Assignee: Lars Hofhansl
HDFS-744-trunk-v5.patch, HDFS-744-trunk.patch, hdfs-744-v2.txt, hdfs-744-v3.txt, hdfs-744.txt

This message is automatically generated by JIRA.


View raw message