{"action":"created","issue":{"url":"https://api.github.com/repos/ipython/ipython/issues/11041","repository_url":"https://api.github.com/repos/ipython/ipython","labels_url":"https://api.github.com/repos/ipython/ipython/issues/11041/labels{/name}","comments_url":"https://api.github.com/repos/ipython/ipython/issues/11041/comments","events_url":"https://api.github.com/repos/ipython/ipython/issues/11041/events","html_url":"https://github.com/ipython/ipython/pull/11041","id":304148714,"node_id":"MDExOlB1bGxSZXF1ZXN0MTc0MjE1MzY3","number":11041,"title":"New input transformation machinery","user":{"login":"takluyver","id":327925,"node_id":"MDQ6VXNlcjMyNzkyNQ==","avatar_url":"https://avatars3.githubusercontent.com/u/327925?v=4","gravatar_id":"","url":"https://api.github.com/users/takluyver","html_url":"https://github.com/takluyver","followers_url":"https://api.github.com/users/takluyver/followers","following_url":"https://api.github.com/users/takluyver/following{/other_user}","gists_url":"https://api.github.com/users/takluyver/gists{/gist_id}","starred_url":"https://api.github.com/users/takluyver/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/takluyver/subscriptions","organizations_url":"https://api.github.com/users/takluyver/orgs","repos_url":"https://api.github.com/users/takluyver/repos","events_url":"https://api.github.com/users/takluyver/events{/privacy}","received_events_url":"https://api.github.com/users/takluyver/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/ipython/ipython/milestones/43","html_url":"https://github.com/ipython/ipython/milestone/43","labels_url":"https://api.github.com/repos/ipython/ipython/milestones/43/labels","id":2298419,"node_id":"MDk6TWlsZXN0b25lMjI5ODQxOQ==","number":43,"title":"7.0","description":"Version 7.0 of IPython, date not yet scheduled, but we can start adding issues about deprecation and feature removal.","creator":{"login":"Carreau","id":335567,"node_id":"MDQ6VXNlcjMzNTU2Nw==","avatar_url":"https://avatars3.githubusercontent.com/u/335567?v=4","gravatar_id":"","url":"https://api.github.com/users/Carreau","html_url":"https://github.com/Carreau","followers_url":"https://api.github.com/users/Carreau/followers","following_url":"https://api.github.com/users/Carreau/following{/other_user}","gists_url":"https://api.github.com/users/Carreau/gists{/gist_id}","starred_url":"https://api.github.com/users/Carreau/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Carreau/subscriptions","organizations_url":"https://api.github.com/users/Carreau/orgs","repos_url":"https://api.github.com/users/Carreau/repos","events_url":"https://api.github.com/users/Carreau/events{/privacy}","received_events_url":"https://api.github.com/users/Carreau/received_events","type":"User","site_admin":false},"open_issues":7,"closed_issues":42,"state":"open","created_at":"2017-02-03T17:13:35Z","updated_at":"2018-09-02T00:29:40Z","due_on":null,"closed_at":null},"comments":30,"created_at":"2018-03-11T10:57:51Z","updated_at":"2018-09-02T16:42:10Z","closed_at":"2018-09-01T21:32:59Z","author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/ipython/ipython/pulls/11041","html_url":"https://github.com/ipython/ipython/pull/11041","diff_url":"https://github.com/ipython/ipython/pull/11041.diff","patch_url":"https://github.com/ipython/ipython/pull/11041.patch"},"body":"See #10272. This is a major change which I expect to be the trigger for IPython 7.0.\r\n\r\n## Main changes\r\n\r\n* `IPython.core.inputtransformer2` is added, containing the new machinery to transform input before we parse it.\r\n* `IPython.core.inputsplitter` and `IPython.core.inputtransformer` are deprecated. We can't remove them immediately because nbconvert relies on them.\r\n* `IPython.utils.tokenize2` is removed. This was a copy of the stdlib `tokenize` module with a couple of issues patched. One of them ([bpo 12691](https://bugs.python.org/issue12691)) was fixed in Python 3.4.1, which we will require for IPython 7 (see #10833). The other ([bpo 17061](https://bugs.python.org/issue17061)) turned out to be intended behaviour, so I've handled it another way in inputtransformer2.\r\n\r\n## Rationale\r\n\r\nThis will break any code that extends the current input transformation machinery (and the interface was documented), and any change this big is bound to cause some unexpected bugs. So why should we do this?\r\n\r\n* Maintainability: I wrote the current input transformation machinery when I'd just learned about coroutines, and I think I got carried away. I don't think anyone else wants to touch that code, and I'm wary of doing so myself. I don't think I can make the code to do this *simple* - it's just too complex a task - but I believe the new code is more quite a bit *simpler* than what it replaces.\r\n  * Code size is not a good measure of complexity, but FWIW, 550 lines in inputtransformer2 replace 1300 lines in inputsplitter+inputtransformer.\r\n* Weird quirks: I hope that the new framework, which uses tokens to recognise our special syntax, will be more robust against corner cases involving comments, multiline strings, etc. #10972 appears to be fixed. Hopefully #8987 is too, though I can't currently reproduce it on master.\r\n* Performance: despite some improvements to the existing code in #10898, it still gets pathologically slow with long enough code, as discussed in #11026. That latter issue contains a sample of ~1500 lines of plain Python code (a giant dictionary literal), which takes ~20s to run through inputsplitter on master before it can be executed. The new transformation machinery in this PR processes it in about 30ms, a speedup of three orders of magnitude. :sunglasses: \r\n\r\n## Limitations\r\n\r\n* Worst case performance: Every piece of special syntax requiring token based transformation (`%line_magics`, `!system commands`, `help?`, etc.) results in another tokenisation pass after the transformation. So a large code cell with many instances of special syntax could take longer to parse than it currently does - although the current code isn't very efficient either (see above). I haven't tried to measure this. My intuition is that large cells tend to be regular Python code, as in #11026, in which case the new implementation can comfortably beat the old one.\r\n* Extensibility of 'is this complete?' checking: the old code allows transformers to 'accumulate' code to indicate that they'd like further lines of input from the user. The new system currently has no equivalent. It's a source of complexity which I'm not sure if anyone actually needs, so I propose we skip it for now and see how many people complain."},"comment":{"url":"https://api.github.com/repos/ipython/ipython/issues/comments/417943369","html_url":"https://github.com/ipython/ipython/pull/11041#issuecomment-417943369","issue_url":"https://api.github.com/repos/ipython/ipython/issues/11041","id":417943369,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNzk0MzM2OQ==","user":{"login":"Carreau","id":335567,"node_id":"MDQ6VXNlcjMzNTU2Nw==","avatar_url":"https://avatars3.githubusercontent.com/u/335567?v=4","gravatar_id":"","url":"https://api.github.com/users/Carreau","html_url":"https://github.com/Carreau","followers_url":"https://api.github.com/users/Carreau/followers","following_url":"https://api.github.com/users/Carreau/following{/other_user}","gists_url":"https://api.github.com/users/Carreau/gists{/gist_id}","starred_url":"https://api.github.com/users/Carreau/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Carreau/subscriptions","organizations_url":"https://api.github.com/users/Carreau/orgs","repos_url":"https://api.github.com/users/Carreau/repos","events_url":"https://api.github.com/users/Carreau/events{/privacy}","received_events_url":"https://api.github.com/users/Carreau/received_events","type":"User","site_admin":false},"created_at":"2018-09-02T16:42:10Z","updated_at":"2018-09-02T16:42:10Z","author_association":"MEMBER","body":"No, but the bot can read inside HTML comments, that aren't shown in the UI!\n\nOn Sun, Sep 2, 2018, 00:27 Thomas Kluyver <notifications@github.com> wrote:\n\n> :-P\n>\n> Does saying the bots name and the word 'excited' in the same comment\n> trigger that?\n>\n> —\n> You are receiving this because you modified the open/close state.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/ipython/ipython/pull/11041#issuecomment-417910867>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/AAUez322zIhNiwJpySGHfGI1iHKc5YFoks5uW4hjgaJpZM4SlpR3>\n> .\n>\n"}}
{"actor":{"display_login":"Carreau"}}