{"action":"opened","issue":{"url":"https://api.github.com/repos/tskit-dev/msprime/issues/456","repository_url":"https://api.github.com/repos/tskit-dev/msprime","labels_url":"https://api.github.com/repos/tskit-dev/msprime/issues/456/labels{/name}","comments_url":"https://api.github.com/repos/tskit-dev/msprime/issues/456/comments","events_url":"https://api.github.com/repos/tskit-dev/msprime/issues/456/events","html_url":"https://github.com/tskit-dev/msprime/issues/456","id":313830772,"number":456,"title":"Clarify encoding of binary metadata for binary input/output","user":{"login":"petrelharp","id":1046249,"avatar_url":"https://avatars2.githubusercontent.com/u/1046249?v=4","gravatar_id":"","url":"https://api.github.com/users/petrelharp","html_url":"https://github.com/petrelharp","followers_url":"https://api.github.com/users/petrelharp/followers","following_url":"https://api.github.com/users/petrelharp/following{/other_user}","gists_url":"https://api.github.com/users/petrelharp/gists{/gist_id}","starred_url":"https://api.github.com/users/petrelharp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petrelharp/subscriptions","organizations_url":"https://api.github.com/users/petrelharp/orgs","repos_url":"https://api.github.com/users/petrelharp/repos","events_url":"https://api.github.com/users/petrelharp/events{/privacy}","received_events_url":"https://api.github.com/users/petrelharp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-12T18:03:15Z","updated_at":"2018-04-12T18:03:15Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Two things here.  First, I want to check whether arbitrary bytes are allowed in `metadata`, `derived_state`, and `ancestral_state` columns, for `tskit` to deal with them. (I believe they are now, but want to check everyone is on board with this.)\r\n\r\nSecond, I run into issues when trying to actually do this. We are storing in `derived_state` a sequence of ints that indicate a set of slim's mutations, just packed into the `char *`, as done [here](https://github.com/jgallowa07/SLiM/blob/f7dc259dc07dab15aa9d5ee652d4e4cdf287dfce/core/slim_sim.cpp#L4029-L4030).  Naively, i was hoping that I could `table_collection_dump()` these tables, then read them into python and do things. But, when I try do many thing with the tree sequence in python, I get `utf-8` errors, like:\r\n```\r\n>>> ts = msprime.load(\"test_output/test_output.treeseq\")\r\n>>> vv = ts.variants()\r\n>>> v = next(vv)\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 1, in <module>\r\n  File \"/home/peter/.local/lib/python3.6/site-packages/msprime/trees.py\", line 1995, in variants\r\n    for site_id, genotypes, alleles in iterator:\r\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xdc in position 0: invalid continuation byte\r\n```\r\nI have not yet tracked down where this assumption comes in, but I don't understand the big picture. Could you clarify, @jeromekelleher?"}}
{"actor":{"display_login":"petrelharp"}}