{"action":"created","issue":{"url":"https://api.github.com/repos/edsu/pymarc/issues/18","repository_url":"https://api.github.com/repos/edsu/pymarc","labels_url":"https://api.github.com/repos/edsu/pymarc/issues/18/labels{/name}","comments_url":"https://api.github.com/repos/edsu/pymarc/issues/18/comments","events_url":"https://api.github.com/repos/edsu/pymarc/issues/18/events","html_url":"https://github.com/edsu/pymarc/issues/18","id":2078568,"number":18,"title":"unicode leader fields","user":{"login":"gugek","id":223778,"avatar_url":"https://avatars.githubusercontent.com/u/223778?v=3","gravatar_id":"","url":"https://api.github.com/users/gugek","html_url":"https://github.com/gugek","followers_url":"https://api.github.com/users/gugek/followers","following_url":"https://api.github.com/users/gugek/following{/other_user}","gists_url":"https://api.github.com/users/gugek/gists{/gist_id}","starred_url":"https://api.github.com/users/gugek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gugek/subscriptions","organizations_url":"https://api.github.com/users/gugek/orgs","repos_url":"https://api.github.com/users/gugek/repos","events_url":"https://api.github.com/users/gugek/events{/privacy}","received_events_url":"https://api.github.com/users/gugek/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2011-10-28T13:33:53Z","updated_at":"2016-08-11T12:39:07Z","closed_at":"2016-08-11T12:39:07Z","body":"I ran into an issue with serializing from marc_xml into marc21 transmission. I get a UnicodeDecode error if any of my data is beyond the default encoding (ASCII) because there is a mix of types. This could also be an issue if someone modifies the leader through record.leader and somehow puts in a unicode object instead of an ordinary string object. It doesn't show up in the other as_marc tests because there isn't any writing out of both unicode data and leaders.\r\n\r\nI've got a test at: https://gist.github.com/1322248\r\n\r\n```python\r\ndef test_writing_unicode_leader(self):\r\n    \"\"\"Test serializing a record with a unicode leader\"\"\"\r\n    record = Record()\r\n    record.add_field(Field(245, ['1', '0'], ['a', unichr(0x1234)]))\r\n    record.leader = u'         a              ' # unicode leader here.\r\n    writer = MARCWriter(open('test/foo', 'w'))\r\n    writer.write(record)\r\n    writer.close()\r\n\r\n    reader = MARCReader(open('test/foo'), to_unicode=True)\r\n    record = reader.next()\r\n    self.assertEqual(record['245']['a'], unichr(0x1234))\r\n\r\n    os.remove('test/foo')\r\n```\r\n\r\nHere is my traceback:\r\n\r\n    Traceback (most recent call last):\r\n      File \"/.../marc8.py\", line 69, in test_writing_unicode_leader\r\n        writer.write(record)\r\n      File \"/...\", line 40, in write\r\n        self.file_handle.write(record.as_marc())\r\n      File \"/...\", line 249, in as_marc\r\n        return self.leader + directory + fields\r\n    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 4: ordinal not in range(128)\r\n\r\nI can submit a patch, but my problem is I am kind of mystified by where I should be dealing with this:\r\n\r\n+ should it be in the ingest in marc_xml.py, I could sort of backpedal the leader into string.\r\n+ should it be in the as_marc() serialization, where we can decode the leader if it is unicode.\r\n+ or should it be pulled into the larger unicode issue that some pymarc conversation has talked about; adding __unicode__ to the field and record objects."},"comment":{"url":"https://api.github.com/repos/edsu/pymarc/issues/comments/239148928","html_url":"https://github.com/edsu/pymarc/issues/18#issuecomment-239148928","issue_url":"https://api.github.com/repos/edsu/pymarc/issues/18","id":239148928,"user":{"login":"Wooble","id":223942,"avatar_url":"https://avatars.githubusercontent.com/u/223942?v=3","gravatar_id":"","url":"https://api.github.com/users/Wooble","html_url":"https://github.com/Wooble","followers_url":"https://api.github.com/users/Wooble/followers","following_url":"https://api.github.com/users/Wooble/following{/other_user}","gists_url":"https://api.github.com/users/Wooble/gists{/gist_id}","starred_url":"https://api.github.com/users/Wooble/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Wooble/subscriptions","organizations_url":"https://api.github.com/users/Wooble/orgs","repos_url":"https://api.github.com/users/Wooble/repos","events_url":"https://api.github.com/users/Wooble/events{/privacy}","received_events_url":"https://api.github.com/users/Wooble/received_events","type":"User","site_admin":false},"created_at":"2016-08-11T12:39:07Z","updated_at":"2016-08-11T12:39:07Z","body":"This test now passes on python 2 (the whole concept doesn't make sense in python 3; the leader needs to be bytes and it shouldn't be possible for a user of the library to accidentally stick in the wrong type because of the strict bytes/str separation in py3."}}
{"actor":{"display_login":"Wooble"}}