{"action":"created","issue":{"url":"https://api.github.com/repos/cylc/cylc/issues/2825","repository_url":"https://api.github.com/repos/cylc/cylc","labels_url":"https://api.github.com/repos/cylc/cylc/issues/2825/labels{/name}","comments_url":"https://api.github.com/repos/cylc/cylc/issues/2825/comments","events_url":"https://api.github.com/repos/cylc/cylc/issues/2825/events","html_url":"https://github.com/cylc/cylc/pull/2825","id":375272679,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI2NzU3Mzg0","number":2825,"title":"WIP proof of concept replacing CherryPy by Tornado","user":{"login":"kinow","id":304786,"node_id":"MDQ6VXNlcjMwNDc4Ng==","avatar_url":"https://avatars1.githubusercontent.com/u/304786?v=4","gravatar_id":"","url":"https://api.github.com/users/kinow","html_url":"https://github.com/kinow","followers_url":"https://api.github.com/users/kinow/followers","following_url":"https://api.github.com/users/kinow/following{/other_user}","gists_url":"https://api.github.com/users/kinow/gists{/gist_id}","starred_url":"https://api.github.com/users/kinow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kinow/subscriptions","organizations_url":"https://api.github.com/users/kinow/orgs","repos_url":"https://api.github.com/users/kinow/repos","events_url":"https://api.github.com/users/kinow/events{/privacy}","received_events_url":"https://api.github.com/users/kinow/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"kinow","id":304786,"node_id":"MDQ6VXNlcjMwNDc4Ng==","avatar_url":"https://avatars1.githubusercontent.com/u/304786?v=4","gravatar_id":"","url":"https://api.github.com/users/kinow","html_url":"https://github.com/kinow","followers_url":"https://api.github.com/users/kinow/followers","following_url":"https://api.github.com/users/kinow/following{/other_user}","gists_url":"https://api.github.com/users/kinow/gists{/gist_id}","starred_url":"https://api.github.com/users/kinow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kinow/subscriptions","organizations_url":"https://api.github.com/users/kinow/orgs","repos_url":"https://api.github.com/users/kinow/repos","events_url":"https://api.github.com/users/kinow/events{/privacy}","received_events_url":"https://api.github.com/users/kinow/received_events","type":"User","site_admin":false},"assignees":[{"login":"kinow","id":304786,"node_id":"MDQ6VXNlcjMwNDc4Ng==","avatar_url":"https://avatars1.githubusercontent.com/u/304786?v=4","gravatar_id":"","url":"https://api.github.com/users/kinow","html_url":"https://github.com/kinow","followers_url":"https://api.github.com/users/kinow/followers","following_url":"https://api.github.com/users/kinow/following{/other_user}","gists_url":"https://api.github.com/users/kinow/gists{/gist_id}","starred_url":"https://api.github.com/users/kinow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kinow/subscriptions","organizations_url":"https://api.github.com/users/kinow/orgs","repos_url":"https://api.github.com/users/kinow/repos","events_url":"https://api.github.com/users/kinow/events{/privacy}","received_events_url":"https://api.github.com/users/kinow/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/cylc/cylc/milestones/5","html_url":"https://github.com/cylc/cylc/milestone/5","labels_url":"https://api.github.com/repos/cylc/cylc/milestones/5/labels","id":262614,"node_id":"MDk6TWlsZXN0b25lMjYyNjE0","number":5,"title":"soon","description":"A few months in the future. Top priority issues and easy fixes should initially be marked against this milestone, and then moved to *next-release* if work progresses quickly enough.","creator":{"login":"hjoliver","id":2362137,"node_id":"MDQ6VXNlcjIzNjIxMzc=","avatar_url":"https://avatars0.githubusercontent.com/u/2362137?v=4","gravatar_id":"","url":"https://api.github.com/users/hjoliver","html_url":"https://github.com/hjoliver","followers_url":"https://api.github.com/users/hjoliver/followers","following_url":"https://api.github.com/users/hjoliver/following{/other_user}","gists_url":"https://api.github.com/users/hjoliver/gists{/gist_id}","starred_url":"https://api.github.com/users/hjoliver/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hjoliver/subscriptions","organizations_url":"https://api.github.com/users/hjoliver/orgs","repos_url":"https://api.github.com/users/hjoliver/repos","events_url":"https://api.github.com/users/hjoliver/events{/privacy}","received_events_url":"https://api.github.com/users/hjoliver/received_events","type":"User","site_admin":false},"open_issues":47,"closed_issues":0,"state":"open","created_at":"2013-02-08T07:07:13Z","updated_at":"2018-10-31T14:54:46Z","due_on":"2018-12-31T08:00:00Z","closed_at":null},"comments":13,"created_at":"2018-10-30T00:39:48Z","updated_at":"2018-10-31T19:30:53Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/cylc/cylc/pulls/2825","html_url":"https://github.com/cylc/cylc/pull/2825","diff_url":"https://github.com/cylc/cylc/pull/2825.diff","patch_url":"https://github.com/cylc/cylc/pull/2825.patch"},"body":"This is a work-in-progress proof of concept for replacing our existing CherryPy by Tornado.\r\n\r\nThis PoC replaces only the Suite Server Program dependency to cherrypy, without any work on the new Web GUI, except that the Suite Server Program would now be using the same framework as the new Web GUI.\r\n\r\n## Notes\r\n\r\n- The digest auth is provided by [tornado-http-auth](https://github.com/gvalkov/tornado-http-auth/). Tornado itself doesn't provide it. That library uses MD5 only, no SHA1\r\n- SSL appears to work OK, as well as the access to contact files, and user creentials\r\n- Travis-CI is not able to run the tests due to SSL issues with Python 2.7.14. This current version of Tornado (5.1.1) is supposedly to work with Python 2.7.9+, so maybe kicking Travis a few more times we may find a set of settings for `.travis.yml` that works (not considering testing with Python 2.6, as Tornado 6.x will drop the 2.x support)\r\n- I managed to run `etc/examples/cycling/five` and the complete test-battery for `tests/broadcast/`, but **help testing it would be appreciated** as per item above Travis-CI is not building it yet\r\n- The Tornado [_handlers_](https://www.tornadoweb.org/en/stable/web.html#request-handlers) are organized per \"category\" (e.g. _suite, task, broadcast, ..._), using the same approach from JupyterHub\r\n- Tornado requires a couple dependencies for 2.7 (singledispatch & another one I can't remember the name right now). And the requirements may change depending on the Python version and which features we use. This is true too for other Python frameworks. As we move to Python 3, having at least `requirements.txt` and managing dependencies externally is the minimum requirement for maintainability IMHO. `setup.py`, `tox.ini`, etc, would be nice too, but we can start by just using pip to manage dependencies with requirements.txt if it's better\r\n- CherryPy works with threads. These threads run in parallel with the main thread. In the main thread, `scheduler.py` runs the main loop for the suite server program, checking the database, task pool, etc. On this branch, **Tornado owns the main loop**, and the old-main-loop is now executed in a `PeriodicCallback`, every 1000 ms. So every second Tornado gives the suite server program main loop a chance to run in Tornado's main loop, while this main loop is also busy serving requests.\r\n- I have not used any async/coroutines/decorators in the code yet, as I would first need to learn if there is any blocking calls in the code called by Tornado (I believe the calls to a database like [postgres](https://github.com/tornadoweb/tornado/wiki/Links#async-client-libraries-built-on-tornadoioloop) can be done asynchronously, but not sure about sqlite). But I believe we can probably get some performance enhancements after doing that (I prefer to focus on passing all tests first)\r\n\r\n## Benchmarking\r\n\r\nAssuming this PoC actually works OK for all tests (I'll start the test-battery in a few hours on my local computer), my idea is to find some suites to profile and get a benchmark of Cylc latest running Cherry Py on Python 2.\r\n\r\nThen take another benchmark with Cylc on this branch (rebased) running Tornado 5.x on Python 2.\r\n\r\nFinally, we can update to Python 3, and get a final benchmark, showing whether the performance enhancements - if any, but I'm a half-full glass type of guy anyway - that we got came from Python 2 to Python 3, or from Tornado adoption.\r\n\r\n---\r\n\r\nHopefully this will prove useful soon, once we start having to work more on the new Web GUI and move away from CherryPy.\r\n\r\nFeel free to check out this branch, give it a spin, and report any issues or suggestions (or feel free to even update it with commits if I take too long to reply).\r\n\r\nCheers\r\nBruno"},"comment":{"url":"https://api.github.com/repos/cylc/cylc/issues/comments/434818138","html_url":"https://github.com/cylc/cylc/pull/2825#issuecomment-434818138","issue_url":"https://api.github.com/repos/cylc/cylc/issues/2825","id":434818138,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNDgxODEzOA==","user":{"login":"kinow","id":304786,"node_id":"MDQ6VXNlcjMwNDc4Ng==","avatar_url":"https://avatars1.githubusercontent.com/u/304786?v=4","gravatar_id":"","url":"https://api.github.com/users/kinow","html_url":"https://github.com/kinow","followers_url":"https://api.github.com/users/kinow/followers","following_url":"https://api.github.com/users/kinow/following{/other_user}","gists_url":"https://api.github.com/users/kinow/gists{/gist_id}","starred_url":"https://api.github.com/users/kinow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kinow/subscriptions","organizations_url":"https://api.github.com/users/kinow/orgs","repos_url":"https://api.github.com/users/kinow/repos","events_url":"https://api.github.com/users/kinow/events{/privacy}","received_events_url":"https://api.github.com/users/kinow/received_events","type":"User","site_admin":false},"created_at":"2018-10-31T19:30:53Z","updated_at":"2018-10-31T19:30:53Z","author_association":"MEMBER","body":">We may be able to replace all the sleep and loop management logic in the current main loop with asyncio? But it may be quite involved.\r\n\r\n+1, I think in Python 3.5+ with Tornado, the Tornado main loop is actually a wrapper for the asyncio loop. And it's possible to use asyncio with Tornado, or just Tornado (from an API point-of-view)\r\n\r\n>Other things to note: The current logic for writing to the SQLite files are fully synchronous. This is important for the primary file under ~/cylc-run/SUITE/.service/db to ensure that the suite state matches on unscheduled restart. If we need to design an asynchronous system, we need to bear this in mind. The secondary SQLite file under ~/cylc-run/SUITE/log/db is less important - as it is just a copy.\r\n\r\nI know SQLite does some sort of asynchronous/concurrency access control (https://www.sqlite.org/threadsafe.html) depending on how you compile and initialize it. But haven't thought much about how it will play with coroutines. I think the same said about threads for sqlite applies here, but could be wrong.\r\n\r\nMaybe replicating just the current logic from `scheduler.py`'s main loop to avoid always sleeping 1000ms with Tornado/asyncio would be enough for now. "}}
{"actor":{"display_login":"kinow"}}