{"action":"created","issue":{"url":"https://api.github.com/repos/PyTables/PyTables/issues/692","repository_url":"https://api.github.com/repos/PyTables/PyTables","labels_url":"https://api.github.com/repos/PyTables/PyTables/issues/692/labels{/name}","comments_url":"https://api.github.com/repos/PyTables/PyTables/issues/692/comments","events_url":"https://api.github.com/repos/PyTables/PyTables/issues/692/events","html_url":"https://github.com/PyTables/PyTables/issues/692","id":321472227,"number":692,"title":"Table.append problem","user":{"login":"jsancho-gpl","id":11912854,"avatar_url":"https://avatars3.githubusercontent.com/u/11912854?v=4","gravatar_id":"","url":"https://api.github.com/users/jsancho-gpl","html_url":"https://github.com/jsancho-gpl","followers_url":"https://api.github.com/users/jsancho-gpl/followers","following_url":"https://api.github.com/users/jsancho-gpl/following{/other_user}","gists_url":"https://api.github.com/users/jsancho-gpl/gists{/gist_id}","starred_url":"https://api.github.com/users/jsancho-gpl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsancho-gpl/subscriptions","organizations_url":"https://api.github.com/users/jsancho-gpl/orgs","repos_url":"https://api.github.com/users/jsancho-gpl/repos","events_url":"https://api.github.com/users/jsancho-gpl/events{/privacy}","received_events_url":"https://api.github.com/users/jsancho-gpl/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-09T08:11:21Z","updated_at":"2018-05-09T08:13:51Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Original report on [Users’ Mailing List](https://groups.google.com/forum/#!topic/pytables-users/6_9yFxySRL0) by Thomas Marsh.\r\n\r\nI am having a problem I just can't solve. I am trying to concatenate a large number of tables each with the same set of columns, which when combined will add to ~1.6 billion rows. I am reading them in one by one and using tables.Table.append to add each one in to the growing table. However when I use tables.Table.append, the data of all but the initial table are corrupted. Here is a stripped down version which shows the problem. If someone can see what I am doing wrong I would be very grateful. I am running tables 3.2.0 under python 3.4.6.\r\n\r\nfirst the code:\r\n```\r\nimport numpy as np\r\nimport tables\r\n\r\n# Construct a numpy.recarray with one row of two elements\r\n\r\nrow1 = [(3572145, 5.46789e5)]\r\ndt = np.dtype([('a', '>i4'), ('b', '>f4')])\r\narr = np.rec.array(row1, dt)\r\n\r\n# print it\r\nprint('arr =',arr)\r\n\r\n# open a tables file\r\nh5file = tables.open_file('bug.h5',mode='w',title='Bug test')\r\n\r\n# add a table based on the recarray just created\r\ntable = h5file.create_table(h5file.root, 'bug', arr, 'bug')\r\n\r\n# append the same array to create a second row\r\ntable.append(arr)\r\ntable.flush()\r\n\r\n# close the table\r\nh5file.close()\r\n\r\n# open it\r\nh5file = tables.open_file('bug.h5')\r\n\r\n# print the 'a' and 'b' columns: they should each have a pair of identical values\r\n# since the same row was repeated twice \r\ntab = h5file.root.bug\r\nprint('a =',tab.col('a'))\r\nprint('b =',tab.col('b'))\r\n\r\nh5file.close()\r\n```\r\nand then the output:\r\n```\r\narr = [(3572145,  546789.)]\r\na = [    3572145 -1316932096]\r\nb = [  5.46789000e+05   1.70470369e+10]\r\n```\r\nwhy isn't a = [3572145 3572145], and similarly why isn't b = [ 5.46789000e+05 5.46789000e+05]"},"comment":{"url":"https://api.github.com/repos/PyTables/PyTables/issues/comments/387659327","html_url":"https://github.com/PyTables/PyTables/issues/692#issuecomment-387659327","issue_url":"https://api.github.com/repos/PyTables/PyTables/issues/692","id":387659327,"user":{"login":"jsancho-gpl","id":11912854,"avatar_url":"https://avatars3.githubusercontent.com/u/11912854?v=4","gravatar_id":"","url":"https://api.github.com/users/jsancho-gpl","html_url":"https://github.com/jsancho-gpl","followers_url":"https://api.github.com/users/jsancho-gpl/followers","following_url":"https://api.github.com/users/jsancho-gpl/following{/other_user}","gists_url":"https://api.github.com/users/jsancho-gpl/gists{/gist_id}","starred_url":"https://api.github.com/users/jsancho-gpl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsancho-gpl/subscriptions","organizations_url":"https://api.github.com/users/jsancho-gpl/orgs","repos_url":"https://api.github.com/users/jsancho-gpl/repos","events_url":"https://api.github.com/users/jsancho-gpl/events{/privacy}","received_events_url":"https://api.github.com/users/jsancho-gpl/received_events","type":"User","site_admin":false},"created_at":"2018-05-09T08:13:51Z","updated_at":"2018-05-09T08:13:51Z","author_association":"CONTRIBUTOR","body":"You are using big-endian here, but look at what happens when you create the table:\r\n```\r\nIn [55]: dt\r\nOut[55]: dtype([('a', '>i4'), ('b', '>f4')])\r\n\r\nIn [56]: tab._v_dtype\r\nOut[56]: dtype([('a', '<i4'), ('b', '<f4')])\r\n```\r\nPyTables transforms datatypes to little-endian. It looks that the numbers aren't modified when you populate the table directly with create_table, but append method is affected undoubtedly.\r\n\r\nIf you create your array using little-endian, it runs without problem.\r\n\r\nProbably a fix is needed for the ```append``` method."}}
{"actor":{"display_login":"jsancho-gpl"}}