{"action":"created","issue":{"url":"https://api.github.com/repos/labrad/pylabrad/issues/220","repository_url":"https://api.github.com/repos/labrad/pylabrad","labels_url":"https://api.github.com/repos/labrad/pylabrad/issues/220/labels{/name}","comments_url":"https://api.github.com/repos/labrad/pylabrad/issues/220/comments","events_url":"https://api.github.com/repos/labrad/pylabrad/issues/220/events","html_url":"https://github.com/labrad/pylabrad/issues/220","id":135583212,"number":220,"title":"slow datavault saving","user":{"login":"roboguy222","id":6502986,"avatar_url":"https://avatars.githubusercontent.com/u/6502986?v=3","gravatar_id":"","url":"https://api.github.com/users/roboguy222","html_url":"https://github.com/roboguy222","followers_url":"https://api.github.com/users/roboguy222/followers","following_url":"https://api.github.com/users/roboguy222/following{/other_user}","gists_url":"https://api.github.com/users/roboguy222/gists{/gist_id}","starred_url":"https://api.github.com/users/roboguy222/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/roboguy222/subscriptions","organizations_url":"https://api.github.com/users/roboguy222/orgs","repos_url":"https://api.github.com/users/roboguy222/repos","events_url":"https://api.github.com/users/roboguy222/events{/privacy}","received_events_url":"https://api.github.com/users/roboguy222/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2016-02-22T23:55:57Z","updated_at":"2016-02-23T22:19:35Z","closed_at":null,"body":"The [datavault server](https://github.com/martinisgroup/servers/blob/master/data_vault.py) is really really slow when reading or writing any data larger than 100k or 1M points (not that big if adding a waveform or something).  I have been performing some tests on it, and think I have narrowed down the problem to the actual sending of the packets, as opposed to the processing or writing of the data.  To do this, I wrote a simple [test script](https://github.com/McDermott-Group/servers/blob/datavaultTrials/dataVaultTestScript.py) to measure the time it took to send the data to a dummy (empty) setting, to process the data once the saving setting was called, and to call the `addData()` method at the end.  \r\n\r\nI of course also had to edit the server.py datavault file, but the only things I did were a) add a dummy setting, b) add a few time.time() calls to (crudely) time the functions, and c) return those times.\r\n\r\nA summary of the results is found below.  The OoM is the Order of Magnitude of the number of data points, where OoM=6 means I sent 10^6 points.  I suspect that the bottleneck comes from the flattening of data, but it also could be the actual sending.  Have you run into this?  Any ideas what is causing the slowness?\r\n\r\n```\r\nADD timing:\r\nOoM        Send            Process         Write\r\n0          0.002000        0.000000        0.001000\r\n1          0.001000        0.000000        0.001000\r\n2          0.002000        0.000000        0.001000\r\n3          0.007000        0.000000        0.001000\r\n4          0.048000        0.001000        0.001000\r\n5          0.465000        0.001000        0.070000\r\n6          6.352000        0.009000        0.078000\r\nADD EX timing:\r\nOoM        Send            Process         Write\r\n0          0.002000        0.001000        0.000000\r\n1          0.002000        0.000000        0.001000\r\n2          0.003000        0.001000        0.000000\r\n3          0.007000        0.005000        0.001000\r\n4          0.050000        0.050000        0.001000\r\n5          0.476000        0.490000        0.009000\r\n6          6.401000        4.797000        0.081000\r\nADD EX T timing:\r\nOoM        Send            Process         Write\r\n0          0.002000        0.000000        0.001000\r\n1          0.002000        0.000000        0.001000\r\n2          0.003000        0.001000        0.000000\r\n3          0.007000        0.000000        0.001000\r\n4          0.050000        0.001000        0.001000\r\n5          0.484000        0.010000        0.009000\r\n6          6.696000        0.192000        0.077000\r\n```"},"comment":{"url":"https://api.github.com/repos/labrad/pylabrad/issues/comments/187941339","html_url":"https://github.com/labrad/pylabrad/issues/220#issuecomment-187941339","issue_url":"https://api.github.com/repos/labrad/pylabrad/issues/220","id":187941339,"user":{"login":"maffoo","id":129268,"avatar_url":"https://avatars.githubusercontent.com/u/129268?v=3","gravatar_id":"","url":"https://api.github.com/users/maffoo","html_url":"https://github.com/maffoo","followers_url":"https://api.github.com/users/maffoo/followers","following_url":"https://api.github.com/users/maffoo/following{/other_user}","gists_url":"https://api.github.com/users/maffoo/gists{/gist_id}","starred_url":"https://api.github.com/users/maffoo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maffoo/subscriptions","organizations_url":"https://api.github.com/users/maffoo/orgs","repos_url":"https://api.github.com/users/maffoo/repos","events_url":"https://api.github.com/users/maffoo/events{/privacy}","received_events_url":"https://api.github.com/users/maffoo/received_events","type":"User","site_admin":false},"created_at":"2016-02-23T22:19:35Z","updated_at":"2016-02-23T22:19:35Z","body":"@roboguy222, what version of the manager are you using?"}}