{"action":"created","issue":{"url":"https://api.github.com/repos/SCM-NV/qmworks/issues/36","repository_url":"https://api.github.com/repos/SCM-NV/qmworks","labels_url":"https://api.github.com/repos/SCM-NV/qmworks/issues/36/labels{/name}","comments_url":"https://api.github.com/repos/SCM-NV/qmworks/issues/36/comments","events_url":"https://api.github.com/repos/SCM-NV/qmworks/issues/36/events","html_url":"https://github.com/SCM-NV/qmworks/issues/36","id":182542675,"number":36,"title":"Serialize numpy arrays or store in HDF5","user":{"login":"ridderl","id":2386182,"avatar_url":"https://avatars.githubusercontent.com/u/2386182?v=3","gravatar_id":"","url":"https://api.github.com/users/ridderl","html_url":"https://github.com/ridderl","followers_url":"https://api.github.com/users/ridderl/followers","following_url":"https://api.github.com/users/ridderl/following{/other_user}","gists_url":"https://api.github.com/users/ridderl/gists{/gist_id}","starred_url":"https://api.github.com/users/ridderl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ridderl/subscriptions","organizations_url":"https://api.github.com/users/ridderl/orgs","repos_url":"https://api.github.com/users/ridderl/repos","events_url":"https://api.github.com/users/ridderl/events{/privacy}","received_events_url":"https://api.github.com/users/ridderl/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/SCM-NV/qmworks/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"closed","locked":false,"assignee":{"login":"jhidding","id":3082555,"avatar_url":"https://avatars.githubusercontent.com/u/3082555?v=3","gravatar_id":"","url":"https://api.github.com/users/jhidding","html_url":"https://github.com/jhidding","followers_url":"https://api.github.com/users/jhidding/followers","following_url":"https://api.github.com/users/jhidding/following{/other_user}","gists_url":"https://api.github.com/users/jhidding/gists{/gist_id}","starred_url":"https://api.github.com/users/jhidding/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhidding/subscriptions","organizations_url":"https://api.github.com/users/jhidding/orgs","repos_url":"https://api.github.com/users/jhidding/repos","events_url":"https://api.github.com/users/jhidding/events{/privacy}","received_events_url":"https://api.github.com/users/jhidding/received_events","type":"User","site_admin":false},"assignees":[{"login":"jhidding","id":3082555,"avatar_url":"https://avatars.githubusercontent.com/u/3082555?v=3","gravatar_id":"","url":"https://api.github.com/users/jhidding","html_url":"https://github.com/jhidding","followers_url":"https://api.github.com/users/jhidding/followers","following_url":"https://api.github.com/users/jhidding/following{/other_user}","gists_url":"https://api.github.com/users/jhidding/gists{/gist_id}","starred_url":"https://api.github.com/users/jhidding/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhidding/subscriptions","organizations_url":"https://api.github.com/users/jhidding/orgs","repos_url":"https://api.github.com/users/jhidding/repos","events_url":"https://api.github.com/users/jhidding/events{/privacy}","received_events_url":"https://api.github.com/users/jhidding/received_events","type":"User","site_admin":false},{"login":"felipeZ","id":4108663,"avatar_url":"https://avatars.githubusercontent.com/u/4108663?v=3","gravatar_id":"","url":"https://api.github.com/users/felipeZ","html_url":"https://github.com/felipeZ","followers_url":"https://api.github.com/users/felipeZ/followers","following_url":"https://api.github.com/users/felipeZ/following{/other_user}","gists_url":"https://api.github.com/users/felipeZ/gists{/gist_id}","starred_url":"https://api.github.com/users/felipeZ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/felipeZ/subscriptions","organizations_url":"https://api.github.com/users/felipeZ/orgs","repos_url":"https://api.github.com/users/felipeZ/repos","events_url":"https://api.github.com/users/felipeZ/events{/privacy}","received_events_url":"https://api.github.com/users/felipeZ/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2016-10-12T14:36:26Z","updated_at":"2016-10-19T18:06:22Z","closed_at":"2016-10-19T18:06:22Z","body":"I tried to run this script:\r\n```\r\nh2o = rdkitTools.smiles2plams('O')\r\nh2o_freq = gamess(templates.freq, h2o, job_name = \"freq\", work_dir = \"/home/lars/scr\").hessian\r\ns = Settings()\r\ns.inithess = h2o_freq\r\nh2o_opt = adf(templates.geometry.overlay(s), h2o, job_name = \"opt\")\r\n```\r\n\r\nIt crashed with:\r\n```\r\n....\r\n  File \"/home/lars/workspace/workflowengine/noodles/serial/registry.py\", line 324, in encode\r\n    raise NotImplementedError(msg)\r\nNotImplementedError: Cannot encode [  5.83587488e-01  -3.38289452e-03   9.93129190e-14  -2.97726953e-01\r\n  -2.30248008e-01  -1.47776756e-14  -2.85862141e-01   2.33630549e-01\r\n  -3.73399228e-14  -3.38289452e-03   3.62691600e-01   2.51534904e-14\r\n  -1.53536080e-01  -1.75418715e-01  -2.25427316e-13   1.56919114e-01\r\n  -1.87275345e-01   1.28022593e-13   9.93129190e-14   2.51534904e-14\r\n  -1.58563114e-02  -5.85902854e-14  -5.06539255e-14   7.92743537e-03\r\n  -4.07660017e-14   2.55871713e-14   7.92968842e-03  -2.97726953e-01\r\n  -1.53536080e-01  -5.85902854e-14   3.32297741e-01   1.91303251e-01\r\n   2.97938757e-14  -3.45700387e-02  -3.77674987e-02  -4.11996826e-15\r\n  -2.30248008e-01  -1.75418715e-01  -5.06539255e-14   1.91303251e-01\r\n   1.71549718e-01   1.16616786e-13   3.89449502e-02   3.87034274e-03\r\n  -4.84855212e-14  -1.47776756e-14  -2.25427316e-13   7.92743537e-03\r\n   2.97938757e-14   1.16616786e-13  -6.96379742e-03  -1.50053581e-14\r\n   1.08680426e-13  -9.64044822e-04  -2.85862141e-01   1.56919114e-01\r\n  -4.07660017e-14  -3.45700387e-02   3.89449502e-02  -1.50053581e-14\r\n   3.20433037e-01  -1.95863384e-01   4.13297868e-14   2.33630549e-01\r\n  -1.87275345e-01   2.55871713e-14  -3.77674987e-02   3.87034274e-03\r\n   1.08680426e-13  -1.95863384e-01   1.83406115e-01  -7.96238075e-14\r\n  -3.73399228e-14   1.28022593e-13   7.92968842e-03  -4.11996826e-15\r\n  -4.84855212e-14  -9.64044822e-04   4.13297868e-14  -7.96238075e-14\r\n  -6.96604910e-03]: encoder for type `ndarray` is not implemented.\r\n```\r\nSo we now run into the issue of serializing the hessian.\r\nHere we are back to our discussion on how to handle potentially big data arrays and their storage in HDF5.\r\n\r\nHow about the following solution:\r\nWe could add a serializer in packages.py for numpy arrays. This serializer saves the numpy array to an HDF5 and returns the path of the HDF5.\r\nWhat do you think?  "},"comment":{"url":"https://api.github.com/repos/SCM-NV/qmworks/issues/comments/254893861","html_url":"https://github.com/SCM-NV/qmworks/issues/36#issuecomment-254893861","issue_url":"https://api.github.com/repos/SCM-NV/qmworks/issues/36","id":254893861,"user":{"login":"jhidding","id":3082555,"avatar_url":"https://avatars.githubusercontent.com/u/3082555?v=3","gravatar_id":"","url":"https://api.github.com/users/jhidding","html_url":"https://github.com/jhidding","followers_url":"https://api.github.com/users/jhidding/followers","following_url":"https://api.github.com/users/jhidding/following{/other_user}","gists_url":"https://api.github.com/users/jhidding/gists{/gist_id}","starred_url":"https://api.github.com/users/jhidding/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhidding/subscriptions","organizations_url":"https://api.github.com/users/jhidding/orgs","repos_url":"https://api.github.com/users/jhidding/repos","events_url":"https://api.github.com/users/jhidding/events{/privacy}","received_events_url":"https://api.github.com/users/jhidding/received_events","type":"User","site_admin":false},"created_at":"2016-10-19T18:06:22Z","updated_at":"2016-10-19T18:06:22Z","body":"The basic configuration is now noodles. It should be possible to extend this to include a project name in the hdf5 path. The name of the job will be a bit trickier. Also there's a potential performance issue, since we're computing hashes of potentially large data, and then search the entire file for matches to this hash. One workaround for this last issue is to keep a separate database of the arrays that were serialised. I have some more ideas, but for now this issue is closed I think."}}
{"actor":{"display_login":"jhidding"}}