{"action":"created","issue":{"url":"https://api.github.com/repos/rust-lang-nursery/regex/issues/159","repository_url":"https://api.github.com/repos/rust-lang-nursery/regex","labels_url":"https://api.github.com/repos/rust-lang-nursery/regex/issues/159/labels{/name}","comments_url":"https://api.github.com/repos/rust-lang-nursery/regex/issues/159/comments","events_url":"https://api.github.com/repos/rust-lang-nursery/regex/issues/159/events","html_url":"https://github.com/rust-lang-nursery/regex/issues/159","id":131607554,"number":159,"title":"find_iter positions are erronously confusing","user":{"login":"Fiedzia","id":240435,"avatar_url":"https://avatars.githubusercontent.com/u/240435?v=3","gravatar_id":"","url":"https://api.github.com/users/Fiedzia","html_url":"https://github.com/Fiedzia","followers_url":"https://api.github.com/users/Fiedzia/followers","following_url":"https://api.github.com/users/Fiedzia/following{/other_user}","gists_url":"https://api.github.com/users/Fiedzia/gists{/gist_id}","starred_url":"https://api.github.com/users/Fiedzia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fiedzia/subscriptions","organizations_url":"https://api.github.com/users/Fiedzia/orgs","repos_url":"https://api.github.com/users/Fiedzia/repos","events_url":"https://api.github.com/users/Fiedzia/events{/privacy}","received_events_url":"https://api.github.com/users/Fiedzia/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-02-05T10:32:23Z","updated_at":"2016-02-05T14:13:37Z","closed_at":null,"body":"I'm trying to implement some text processing where I'm removing things matching regexp, unless they are in places marked as protected (so keep protected words untouched). It annoys me to no end, as well as generates huge amount of bugs,\r\nthat find_iter returns byte positions instead of char positions.\r\nThis forces me to maintain my own byte-to-char offset mappings, and its not exactly pleasant experience. Can find_iter_chars be added to the api? (I suppose its to late to change find_iter, though I think byte offsets have no place in text processing anymore)."},"comment":{"url":"https://api.github.com/repos/rust-lang-nursery/regex/issues/comments/180374695","html_url":"https://github.com/rust-lang-nursery/regex/issues/159#issuecomment-180374695","issue_url":"https://api.github.com/repos/rust-lang-nursery/regex/issues/159","id":180374695,"user":{"login":"Fiedzia","id":240435,"avatar_url":"https://avatars.githubusercontent.com/u/240435?v=3","gravatar_id":"","url":"https://api.github.com/users/Fiedzia","html_url":"https://github.com/Fiedzia","followers_url":"https://api.github.com/users/Fiedzia/followers","following_url":"https://api.github.com/users/Fiedzia/following{/other_user}","gists_url":"https://api.github.com/users/Fiedzia/gists{/gist_id}","starred_url":"https://api.github.com/users/Fiedzia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fiedzia/subscriptions","organizations_url":"https://api.github.com/users/Fiedzia/orgs","repos_url":"https://api.github.com/users/Fiedzia/repos","events_url":"https://api.github.com/users/Fiedzia/events{/privacy}","received_events_url":"https://api.github.com/users/Fiedzia/received_events","type":"User","site_admin":false},"created_at":"2016-02-05T14:13:37Z","updated_at":"2016-02-05T14:13:37Z","body":"Thanks for response. My issue is that I work with codepoints. All text operations I perform (insert this, cut out that, check if those characters are in in this set etc.) operate on codepoints (.chars()), and suddenly regexp points me to a byte offset.\r\nThere is .char_at(), but it may be expensive.\r\n\r\nBelow is some bit I've wrote. It replaces part of text unless its part of \r\nsome words I want to keep.\r\n\r\nlet mut buf = \"some text with uncode ąśćółń\".to_owned();\r\n`\r\nlet mut protected_positions:Vec<bool> =  vec![false; buf.chars().count()];\r\n        //regexp.find_iter returns ranges as byte positions, we need this logic to translate \r\n        //those into char indices\r\n        let mut char_positions:Vec<usize> =  vec![0; buf.len()];\r\n        for (ch_idx, (byte_idx, ch)) in buf.char_indices().enumerate() {\r\n            for bi in 0..ch.len_utf8() {\r\n                char_positions[byte_idx + bi] = ch_idx;\r\n            }\r\n        }\r\n        for word in PROTECTED_WORDS.iter() {\r\n            for pos in Regex::new(&quote(word)).unwrap().find_iter(&buf) {\r\n                let (byte_start, byte_end) = pos;\r\n                let (start, end) = (char_positions[byte_start], char_positions[byte_end-1]);\r\n                let mut i = start;\r\n                while i < end {\r\n                     protected_positions[i] = true;\r\n                     i+=1;\r\n                }\r\n            }\r\n        };\r\n        buf = buf.chars().enumerate().map(\r\n            |(idx, ch)| {\r\n                if IGNORE_CHARS.binary_search(&ch).is_ok() && !protected_positions[idx] {' '} else {ch}\r\n            }).collect();\r\n`\r\nThe problem is that:\r\n1. Not being able to get codepoint offsets forces me to calculate them myself.\r\n2. This calculation is extremely error prone. I've got dozens of panics before I've got it right. It might be that I'm new to rust, but after coming from python3,\r\nI don't see any reason why any library operating on text would ever use bytes.\r\nPerhaps the assumption that (all) string are bytes in utf8 is a source of the problem.\r\nCould regex accept my alternative string implementation?\r\n\r\n\r\n> Note that all of the string matching functions in std also return byte offsets!\r\n\r\nI am not a fan of that either. I consider this to be source of many issues with handling text.\r\n\r\n"}}