{"action":"created","issue":{"url":"https://api.github.com/repos/nlplab/brat/issues/1179","repository_url":"https://api.github.com/repos/nlplab/brat","labels_url":"https://api.github.com/repos/nlplab/brat/issues/1179/labels{/name}","comments_url":"https://api.github.com/repos/nlplab/brat/issues/1179/comments","events_url":"https://api.github.com/repos/nlplab/brat/issues/1179/events","html_url":"https://github.com/nlplab/brat/issues/1179","id":157649375,"number":1179,"title":"non-ASCII db for normalization","user":{"login":"pierremarchal","id":6877795,"avatar_url":"https://avatars.githubusercontent.com/u/6877795?v=3","gravatar_id":"","url":"https://api.github.com/users/pierremarchal","html_url":"https://github.com/pierremarchal","followers_url":"https://api.github.com/users/pierremarchal/followers","following_url":"https://api.github.com/users/pierremarchal/following{/other_user}","gists_url":"https://api.github.com/users/pierremarchal/gists{/gist_id}","starred_url":"https://api.github.com/users/pierremarchal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pierremarchal/subscriptions","organizations_url":"https://api.github.com/users/pierremarchal/orgs","repos_url":"https://api.github.com/users/pierremarchal/repos","events_url":"https://api.github.com/users/pierremarchal/events{/privacy}","received_events_url":"https://api.github.com/users/pierremarchal/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-05-31T11:29:20Z","updated_at":"2016-06-01T09:48:19Z","closed_at":null,"body":"I built a db file for normalization -- encoding is UTF-8 -- which looks like this : \r\n`6204\tname:Name:Oncogène jun\tname:Name:jun proto-oncogene\tname:Name:AP-1\tname:Name:JUN\tname:Name:JUN gene\tname:Name:c-Jun`\r\nThat is, it contains some non-ASCII characters (e.g. è).\r\n\r\nI compiled the file in Simstring format using the command: \r\n`python tools/norm_db_init.py /home/me/gene.db`\r\n\r\nAnd get the following message: \r\n`Done, imported 39263 entries (162772 strings), skipped 0 duplicate keys, skipped 0 invalid lines`\r\n\r\nNow, I can use the normalization functionnality as long as the search terms do not contain any non-ASCII character. For instance I can search \"PIK3CA\" but not \"gène\".\r\nChecking my apache log I get:\r\n```\r\nTraceback (most recent call last):\r\n  File \"/var/www/html/brat/server/src/server.py\", line 322, in serve\r\n    return _safe_serve(params, client_ip, client_hostname, cookie_data)\r\n  File \"/var/www/html/brat/server/src/server.py\", line 197, in _safe_serve\r\n    json_dic = dispatch(http_args, client_ip, client_hostname)\r\n  File \"/var/www/html/brat/server/src/dispatch.py\", line 308, in dispatch\r\n    json_dic = action_function(*action_args)\r\n  File \"/var/www/html/brat/server/src/norm.py\", line 429, in norm_search\r\n    return _norm_search_impl(database, name, collection, exactmatch)\r\n  File \"/var/www/html/brat/server/src/norm.py\", line 364, in _norm_search_impl\r\n    0, exactmatch)\r\n  File \"/var/www/html/brat/server/src/norm.py\", line 296, in _norm_search_name_attr\r\n    threshold, True)\r\n  File \"/var/www/html/brat/server/src/simstringdb.py\", line 236, in ssdb_supstring_lookup\r\n    if s in r:\r\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)  \r\n```\r\n\r\nAny idea how I could fix this?\r\nThanx. \r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/nlplab/brat/issues/comments/222945041","html_url":"https://github.com/nlplab/brat/issues/1179#issuecomment-222945041","issue_url":"https://api.github.com/repos/nlplab/brat/issues/1179","id":222945041,"user":{"login":"pierremarchal","id":6877795,"avatar_url":"https://avatars.githubusercontent.com/u/6877795?v=3","gravatar_id":"","url":"https://api.github.com/users/pierremarchal","html_url":"https://github.com/pierremarchal","followers_url":"https://api.github.com/users/pierremarchal/followers","following_url":"https://api.github.com/users/pierremarchal/following{/other_user}","gists_url":"https://api.github.com/users/pierremarchal/gists{/gist_id}","starred_url":"https://api.github.com/users/pierremarchal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pierremarchal/subscriptions","organizations_url":"https://api.github.com/users/pierremarchal/orgs","repos_url":"https://api.github.com/users/pierremarchal/repos","events_url":"https://api.github.com/users/pierremarchal/events{/privacy}","received_events_url":"https://api.github.com/users/pierremarchal/received_events","type":"User","site_admin":false},"created_at":"2016-06-01T09:48:19Z","updated_at":"2016-06-01T09:48:19Z","body":"I think I got a solution (thanx to [this](http://stackoverflow.com/questions/21129020/how-to-fix-unicodedecodeerror-ascii-codec-cant-decode-byte) SO post). I had to edit `server/src/simstringdb.py`: \r\n```\r\n#!/usr/bin/env python\r\n# encoding=utf8\r\n\r\nimport glob\r\nimport os\r\nimport sys\r\nreload(sys)\r\nsys.setdefaultencoding('utf-8')\r\n```"}}
{"actor":{"display_login":"pierremarchal"}}