{"action":"created","issue":{"url":"https://api.github.com/repos/w3c/web-annotation/issues/227","repository_url":"https://api.github.com/repos/w3c/web-annotation","labels_url":"https://api.github.com/repos/w3c/web-annotation/issues/227/labels{/name}","comments_url":"https://api.github.com/repos/w3c/web-annotation/issues/227/comments","events_url":"https://api.github.com/repos/w3c/web-annotation/issues/227/events","html_url":"https://github.com/w3c/web-annotation/issues/227","id":155451230,"number":227,"title":"Reference to text encoding in spec perhaps not appropriate","user":{"login":"nickstenning","id":3602,"avatar_url":"https://avatars.githubusercontent.com/u/3602?v=3","gravatar_id":"","url":"https://api.github.com/users/nickstenning","html_url":"https://github.com/nickstenning","followers_url":"https://api.github.com/users/nickstenning/followers","following_url":"https://api.github.com/users/nickstenning/following{/other_user}","gists_url":"https://api.github.com/users/nickstenning/gists{/gist_id}","starred_url":"https://api.github.com/users/nickstenning/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nickstenning/subscriptions","organizations_url":"https://api.github.com/users/nickstenning/orgs","repos_url":"https://api.github.com/users/nickstenning/repos","events_url":"https://api.github.com/users/nickstenning/events{/privacy}","received_events_url":"https://api.github.com/users/nickstenning/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/w3c/web-annotation/labels/i18n-review","name":"i18n-review","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2016-05-18T09:07:10Z","updated_at":"2016-05-30T15:34:46Z","closed_at":null,"body":"#222 made me aware of the following text in the model spec:\r\n\r\n4.2.4 Text Quote Selector\r\nhttps://www.w3.org/TR/2016/WD-annotation-model-20160331/#text-quote-selector\r\n> The text must be normalized before recording. Thus HTML/XML tags should be removed, character entities should be replaced with the character that they encode, unnecessary whitespace should be normalized, **character encoding should be turned into UTF-8**, and so forth. The normalization routine may be performed automatically by a browser, and other applications should implement the DOM String Comparisons method. This allows the Selector to be used with different encodings and user agents and still have the same semantics and utility.\r\n\r\nIf all selector references are to be w.r.t. codepoint sequences (c.f. #206) then I'm not sure the spec should be referring to text encoding. (Because we're assuming that you're annotating unicode text, not some byte sequence.)"},"comment":{"url":"https://api.github.com/repos/w3c/web-annotation/issues/comments/222516176","html_url":"https://github.com/w3c/web-annotation/issues/227#issuecomment-222516176","issue_url":"https://api.github.com/repos/w3c/web-annotation/issues/227","id":222516176,"user":{"login":"fsasaki","id":1177858,"avatar_url":"https://avatars.githubusercontent.com/u/1177858?v=3","gravatar_id":"","url":"https://api.github.com/users/fsasaki","html_url":"https://github.com/fsasaki","followers_url":"https://api.github.com/users/fsasaki/followers","following_url":"https://api.github.com/users/fsasaki/following{/other_user}","gists_url":"https://api.github.com/users/fsasaki/gists{/gist_id}","starred_url":"https://api.github.com/users/fsasaki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fsasaki/subscriptions","organizations_url":"https://api.github.com/users/fsasaki/orgs","repos_url":"https://api.github.com/users/fsasaki/repos","events_url":"https://api.github.com/users/fsasaki/events{/privacy}","received_events_url":"https://api.github.com/users/fsasaki/received_events","type":"User","site_admin":false},"created_at":"2016-05-30T15:34:46Z","updated_at":"2016-05-30T15:34:46Z","body":"> I also recognize that whitespace normalization would destroy \"layout\" such as represented by pre. I think this is expected. If one wants document fidelity, use text positions and extract the layout, not just the plain text.\r\n\r\nThe tools I am referring to are using text to process content as part of a text analytics pipeline. Text analytics tools as of today only understand plain text. So at some point in the text analytics pipeline you have to get rid of the markup - and have to decide about what to do with the white space. That decision will always be format specific, see the DocBook programlisting example. So my point is: you can describe the steps needed only to some extend. At some point implementations have to look into the specifics of  formats to process. That is why step two at https://github.com/w3c/web-annotation/issues/227#issuecomment-222330988 is hard to formulate as a MUST requirement.\r\nI am coming from an XLIFF extracting and merging point of view - which is the same as text analytics with putting the outcome into the orginal content again (= roundtripping). Specs like XLIFF wisely do not speciffy the details of such processes, but say \"be careful about them\" - and then there are - to some extend, not enough - format specific guidelines how to do this. I am not asking for such guidelines here, just trying to explain how big this pandora box is."}}
{"actor":{"display_login":"fsasaki"}}