{"action":"created","issue":{"url":"https://api.github.com/repos/CATIA-Systems/FMPy/issues/26","repository_url":"https://api.github.com/repos/CATIA-Systems/FMPy","labels_url":"https://api.github.com/repos/CATIA-Systems/FMPy/issues/26/labels{/name}","comments_url":"https://api.github.com/repos/CATIA-Systems/FMPy/issues/26/comments","events_url":"https://api.github.com/repos/CATIA-Systems/FMPy/issues/26/events","html_url":"https://github.com/CATIA-Systems/FMPy/issues/26","id":301121767,"number":26,"title":"Problems with multiprocessing caused by the zipfile module","user":{"login":"wulmer","id":1672353,"avatar_url":"https://avatars2.githubusercontent.com/u/1672353?v=4","gravatar_id":"","url":"https://api.github.com/users/wulmer","html_url":"https://github.com/wulmer","followers_url":"https://api.github.com/users/wulmer/followers","following_url":"https://api.github.com/users/wulmer/following{/other_user}","gists_url":"https://api.github.com/users/wulmer/gists{/gist_id}","starred_url":"https://api.github.com/users/wulmer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wulmer/subscriptions","organizations_url":"https://api.github.com/users/wulmer/orgs","repos_url":"https://api.github.com/users/wulmer/repos","events_url":"https://api.github.com/users/wulmer/events{/privacy}","received_events_url":"https://api.github.com/users/wulmer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"t-sommer","id":2453858,"avatar_url":"https://avatars1.githubusercontent.com/u/2453858?v=4","gravatar_id":"","url":"https://api.github.com/users/t-sommer","html_url":"https://github.com/t-sommer","followers_url":"https://api.github.com/users/t-sommer/followers","following_url":"https://api.github.com/users/t-sommer/following{/other_user}","gists_url":"https://api.github.com/users/t-sommer/gists{/gist_id}","starred_url":"https://api.github.com/users/t-sommer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/t-sommer/subscriptions","organizations_url":"https://api.github.com/users/t-sommer/orgs","repos_url":"https://api.github.com/users/t-sommer/repos","events_url":"https://api.github.com/users/t-sommer/events{/privacy}","received_events_url":"https://api.github.com/users/t-sommer/received_events","type":"User","site_admin":false},"assignees":[{"login":"t-sommer","id":2453858,"avatar_url":"https://avatars1.githubusercontent.com/u/2453858?v=4","gravatar_id":"","url":"https://api.github.com/users/t-sommer","html_url":"https://github.com/t-sommer","followers_url":"https://api.github.com/users/t-sommer/followers","following_url":"https://api.github.com/users/t-sommer/following{/other_user}","gists_url":"https://api.github.com/users/t-sommer/gists{/gist_id}","starred_url":"https://api.github.com/users/t-sommer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/t-sommer/subscriptions","organizations_url":"https://api.github.com/users/t-sommer/orgs","repos_url":"https://api.github.com/users/t-sommer/repos","events_url":"https://api.github.com/users/t-sommer/events{/privacy}","received_events_url":"https://api.github.com/users/t-sommer/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2018-02-28T17:53:12Z","updated_at":"2018-03-01T08:06:15Z","closed_at":null,"author_association":"NONE","body":"I was investigating on a really strange issue today: I was using `fmpy` in our project together with a larger multiprocessing setup (doing parallel simulations) and I encountered a strange error (which was hard to debug due to multiprocess stuff) when running on Linux (funnily, Windows worked). The result was a `BrokenProcessPool` error or a `queue.Full` error. \r\nAfter several hours of slicing away code line by code line, I ended up with the following script:\r\n\r\n```python\r\ndef use_fmpy(*args):\r\n    import fmpy\r\n    return fmpy.read_model_description('bouncingBall_me.fmu') \r\n\r\nif __name__ == \"__main__\":\r\n    import tables  ##### TRIGGER HERE\r\n\r\n    import concurrent.futures as cf\r\n    with cf.ProcessPoolExecutor(max_workers=2) as pool:\r\n        futures = []\r\n        for i in range(10):\r\n            future = pool.submit(use_fmpy, None)\r\n            futures.append(future)\r\n        cf.wait(futures)\r\n```\r\n\r\nWhat made it even harder was the fact that the `import tables` line triggered the error in the multiprocessing subprocesses. With the pytables import I got the error, without pytables, the error wasn't appearing.\r\n\r\nSo I first suspected pytables for being the source of the problem, but after a while I took a closer look at the `read_model_description` function. (BTW I tried out all available fmpy versions) The first lines of this function look like this:\r\n\r\nhttps://github.com/CATIA-Systems/FMPy/blob/35245dd8bf2b8be2a8e750b04bd29df4557d3032/fmpy/model_description.py#L210-L212\r\n\r\nIt uses the `zipfile` module to read the `modelDescription.xml` file from the FMU file. Putting a `return` just before this code block made my example above work in all cases, so it must have had to do with the `zipfile` use.\r\n\r\nGoogling brought immediately a possible issue: [It seems that the `zipfile` module is *not threadsafe*](https://stackoverflow.com/a/22853969/195476). So my idea was to replace the `zipfile` use by a call to `unzip`:\r\n\r\n```python\r\nimport subprocess\r\nroot = etree.fromstring(subprocess.check_output(\r\n    ['unzip', '-p', filename, 'modelDescription.xml'])) \r\n```\r\n\r\nAnd surprise: it worked! No more problems with `BrokenProcessPool` or `queue.Full` errors.\r\n\r\nSo my question here: would it be possible that you replace all the `zipfile` usages by something more robust? I would really love to use `fmpy` instead of `pyfmi`, but this issue cost me already 5 hours of debugging and it is currently a blocker for us."},"comment":{"url":"https://api.github.com/repos/CATIA-Systems/FMPy/issues/comments/369508936","html_url":"https://github.com/CATIA-Systems/FMPy/issues/26#issuecomment-369508936","issue_url":"https://api.github.com/repos/CATIA-Systems/FMPy/issues/26","id":369508936,"user":{"login":"t-sommer","id":2453858,"avatar_url":"https://avatars1.githubusercontent.com/u/2453858?v=4","gravatar_id":"","url":"https://api.github.com/users/t-sommer","html_url":"https://github.com/t-sommer","followers_url":"https://api.github.com/users/t-sommer/followers","following_url":"https://api.github.com/users/t-sommer/following{/other_user}","gists_url":"https://api.github.com/users/t-sommer/gists{/gist_id}","starred_url":"https://api.github.com/users/t-sommer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/t-sommer/subscriptions","organizations_url":"https://api.github.com/users/t-sommer/orgs","repos_url":"https://api.github.com/users/t-sommer/repos","events_url":"https://api.github.com/users/t-sommer/events{/privacy}","received_events_url":"https://api.github.com/users/t-sommer/received_events","type":"User","site_admin":false},"created_at":"2018-03-01T08:06:15Z","updated_at":"2018-03-01T08:06:15Z","author_association":"CONTRIBUTOR","body":"Thanks a lot for digging into this and the detailed description! Can you please try if e32601c solves your problem? It adds a [lock](https://docs.python.org/2/library/threading.html#lock-objects) to synchronize the section that causes the problem so only one thread can access it at a time."}}
{"actor":{"display_login":"t-sommer"}}