{"action":"opened","issue":{"url":"https://api.github.com/repos/nijel/weblate/issues/1166","repository_url":"https://api.github.com/repos/nijel/weblate","labels_url":"https://api.github.com/repos/nijel/weblate/issues/1166/labels{/name}","comments_url":"https://api.github.com/repos/nijel/weblate/issues/1166/comments","events_url":"https://api.github.com/repos/nijel/weblate/issues/1166/events","html_url":"https://github.com/nijel/weblate/issues/1166","id":163831955,"number":1166,"title":"Exception in update_index and rebuild_index: UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 4: ordinal not in range(128)","user":{"login":"ihoru","id":361908,"avatar_url":"https://avatars.githubusercontent.com/u/361908?v=3","gravatar_id":"","url":"https://api.github.com/users/ihoru","html_url":"https://github.com/ihoru","followers_url":"https://api.github.com/users/ihoru/followers","following_url":"https://api.github.com/users/ihoru/following{/other_user}","gists_url":"https://api.github.com/users/ihoru/gists{/gist_id}","starred_url":"https://api.github.com/users/ihoru/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ihoru/subscriptions","organizations_url":"https://api.github.com/users/ihoru/orgs","repos_url":"https://api.github.com/users/ihoru/repos","events_url":"https://api.github.com/users/ihoru/events{/privacy}","received_events_url":"https://api.github.com/users/ihoru/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-07-05T11:45:51Z","updated_at":"2016-07-05T11:45:51Z","closed_at":null,"body":"### Steps to reproduce\r\n```\r\n$ ./manage.py update_index\r\nTraceback (most recent call last):\r\n  File \"/opt/weblate/weblate/trans/search.py\", line 176, in update_index\r\n    update_source_unit_index(writer, unit)\r\n  File \"/opt/weblate/weblate/trans/search.py\", line 102, in update_source_unit_index\r\n    location=force_text(unit.location),\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 1255, in update_document\r\n    IndexWriter.update_document(self, **fields)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 490, in update_document\r\n    self.add_document(**fields)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 1251, in add_document\r\n    self.commit()\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 1229, in commit\r\n    self.writer.commit(**self.commitargs)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 922, in commit\r\n    finalsegments = self._merge_segments(mergetype, optimize, merge)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 827, in _merge_segments\r\n    return mergetype(self, self.segments)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 101, in MERGE_SMALL\r\n    writer.add_reader(reader)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 710, in add_reader\r\n    self.add_postings_to_pool(reader, basedoc, docmap)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 647, in add_postings_to_pool\r\n    for item in items:\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 583, in _process_posts\r\n    for fieldname, text, docnum, weight, vbytes in items:\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/reading.py\", line 427, in iter_postings\r\n    m = self.postings(fieldname, btext)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/reading.py\", line 821, in postings\r\n    matcher = FilterMatcher(matcher, deleted, exclude=True)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/matching/wrappers.py\", line 277, in __init__\r\n    self._find_next()\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/matching/wrappers.py\", line 302, in _find_next\r\n    while child.is_active() and child.id() in ids:\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/codec/whoosh3.py\", line 980, in id\r\n    self._read_ids()\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/codec/whoosh3.py\", line 1082, in _read_ids\r\n    self._read_data()\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/codec/whoosh3.py\", line 1077, in _read_data\r\n    self._data = loads(b)\r\nUnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 4: ordinal not in range(128)\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"./manage.py\", line 31, in <module>\r\n    execute_from_command_line(sys.argv)\r\n  File \"/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py\", line 353, in execute_from_command_line\r\n    utility.execute()\r\n  File \"/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py\", line 345, in execute\r\n    self.fetch_command(subcommand).run_from_argv(self.argv)\r\n  File \"/usr/local/lib/python3.5/dist-packages/django/core/management/base.py\", line 348, in run_from_argv\r\n    self.execute(*args, **cmd_options)\r\n  File \"/usr/local/lib/python3.5/dist-packages/django/core/management/base.py\", line 399, in execute\r\n    output = self.handle(*args, **options)\r\n  File \"/opt/weblate/weblate/trans/management/commands/update_index.py\", line 43, in handle\r\n    self.do_update(options['limit'])\r\n  File \"/opt/weblate/weblate/trans/management/commands/update_index.py\", line 93, in do_update\r\n    update_index(units, source_units)\r\n  File \"/opt/weblate/weblate/trans/search.py\", line 178, in update_index\r\n    writer.close()\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 1217, in close\r\n    self.commit(restart=False)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 1229, in commit\r\n    self.writer.commit(**self.commitargs)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 922, in commit\r\n    finalsegments = self._merge_segments(mergetype, optimize, merge)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 827, in _merge_segments\r\n    return mergetype(self, self.segments)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 101, in MERGE_SMALL\r\n    writer.add_reader(reader)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 710, in add_reader\r\n    self.add_postings_to_pool(reader, basedoc, docmap)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 647, in add_postings_to_pool\r\n    for item in items:\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 583, in _process_posts\r\n    for fieldname, text, docnum, weight, vbytes in items:\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/reading.py\", line 427, in iter_postings\r\n    m = self.postings(fieldname, btext)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/reading.py\", line 821, in postings\r\n    matcher = FilterMatcher(matcher, deleted, exclude=True)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/matching/wrappers.py\", line 277, in __init__\r\n    self._find_next()\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/matching/wrappers.py\", line 302, in _find_next\r\n    while child.is_active() and child.id() in ids:\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/codec/whoosh3.py\", line 980, in id\r\n    self._read_ids()\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/codec/whoosh3.py\", line 1082, in _read_ids\r\n    self._read_data()\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/codec/whoosh3.py\", line 1077, in _read_data\r\n    self._data = loads(b)\r\nUnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 4: ordinal not in range(128)\r\n\r\n\r\n\r\n$ ./manage.py rebuild_index --all\r\nProcessing 0.0%\r\nProcessing 7.5%\r\nProcessing 14.9%\r\nProcessing 22.4%\r\nProcessing 29.9%\r\nProcessing 37.4%\r\nProcessing 44.8%\r\nProcessing 52.3%\r\nProcessing 59.8%\r\nProcessing 67.2%\r\nProcessing 74.7%\r\nProcessing 82.2%\r\nProcessing 89.7%\r\nProcessing 97.1%\r\nOperation completed\r\nTraceback (most recent call last):\r\n  File \"./manage.py\", line 31, in <module>\r\n    execute_from_command_line(sys.argv)\r\n  File \"/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py\", line 353, in execute_from_command_line\r\n    utility.execute()\r\n  File \"/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py\", line 345, in execute\r\n    self.fetch_command(subcommand).run_from_argv(self.argv)\r\n  File \"/usr/local/lib/python3.5/dist-packages/django/core/management/base.py\", line 348, in run_from_argv\r\n    self.execute(*args, **cmd_options)\r\n  File \"/usr/local/lib/python3.5/dist-packages/django/core/management/base.py\", line 399, in execute\r\n    output = self.handle(*args, **options)\r\n  File \"/opt/weblate/weblate/trans/management/commands/rebuild_index.py\", line 89, in handle\r\n    target_writers[code].commit()\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 922, in commit\r\n    finalsegments = self._merge_segments(mergetype, optimize, merge)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 827, in _merge_segments\r\n    return mergetype(self, self.segments)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 101, in MERGE_SMALL\r\n    writer.add_reader(reader)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 710, in add_reader\r\n    self.add_postings_to_pool(reader, basedoc, docmap)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 647, in add_postings_to_pool\r\n    for item in items:\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/writing.py\", line 583, in _process_posts\r\n    for fieldname, text, docnum, weight, vbytes in items:\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/reading.py\", line 427, in iter_postings\r\n    m = self.postings(fieldname, btext)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/reading.py\", line 821, in postings\r\n    matcher = FilterMatcher(matcher, deleted, exclude=True)\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/matching/wrappers.py\", line 277, in __init__\r\n    self._find_next()\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/matching/wrappers.py\", line 302, in _find_next\r\n    while child.is_active() and child.id() in ids:\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/codec/whoosh3.py\", line 980, in id\r\n    self._read_ids()\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/codec/whoosh3.py\", line 1082, in _read_ids\r\n    self._read_data()\r\n  File \"/usr/local/lib/python3.5/dist-packages/whoosh/codec/whoosh3.py\", line 1077, in _read_data\r\n    self._data = loads(b)\r\nUnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 4: ordinal not in range(128)\r\n```\r\n\r\n### Server configuration\r\n```\r\n$ ./manage.py list_versions\r\n * Weblate weblate-2.6-324-g16e3843\r\n * Python 3.5.1+\r\n * Django 1.9.7\r\n * six 1.10.0\r\n * python-social-auth 0.2.19\r\n * Translate Toolkit 2.0.0b2\r\n * Whoosh 2.7.4\r\n * Git 2.7.4\r\n * Pillow (PIL) 1.1.7\r\n * dateutil 2.5.3\r\n * lxml 3.6.0\r\n * django-crispy-forms 1.6.0\r\n * compressor 1.6\r\n * djangorestframework 3.3.3\r\n * pytz 2016.4\r\n * pyuca N/A\r\n * python-bidi 0.4.0\r\n * pyLibravatar N/A\r\n * Database backends: django.db.backends.mysql\r\n```"}}
{"actor":{"display_login":"ihoru"}}