{"action":"created","issue":{"url":"https://api.github.com/repos/davidhalter/jedi/issues/630","repository_url":"https://api.github.com/repos/davidhalter/jedi","labels_url":"https://api.github.com/repos/davidhalter/jedi/issues/630/labels{/name}","comments_url":"https://api.github.com/repos/davidhalter/jedi/issues/630/comments","events_url":"https://api.github.com/repos/davidhalter/jedi/issues/630/events","html_url":"https://github.com/davidhalter/jedi/issues/630","id":109576582,"number":630,"title":"Create a unified tech stack for jedi, rope, RedBaron, yapf, pylint, pep8, and similar tools and libraries","user":{"login":"gotgenes","id":8517,"avatar_url":"https://avatars.githubusercontent.com/u/8517?v=3","gravatar_id":"","url":"https://api.github.com/users/gotgenes","html_url":"https://github.com/gotgenes","followers_url":"https://api.github.com/users/gotgenes/followers","following_url":"https://api.github.com/users/gotgenes/following{/other_user}","gists_url":"https://api.github.com/users/gotgenes/gists{/gist_id}","starred_url":"https://api.github.com/users/gotgenes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gotgenes/subscriptions","organizations_url":"https://api.github.com/users/gotgenes/orgs","repos_url":"https://api.github.com/users/gotgenes/repos","events_url":"https://api.github.com/users/gotgenes/events{/privacy}","received_events_url":"https://api.github.com/users/gotgenes/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/davidhalter/jedi/labels/discussion","name":"discussion","color":"444444"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":35,"created_at":"2015-10-02T21:38:48Z","updated_at":"2016-07-03T03:18:43Z","closed_at":null,"body":"Currently there exists duplication of effort around tooling for parsing, automated manipulation, and analyzing Python code. Each project in this space faces similar fundamental problems of requiring a data representation of the Python code of interest, most commonly represented as a syntax tree. Here is a list, by no means exhaustive, of tools and libraries that face the problem of needing a representation of Python code capable of inspection and possibly even manipulation:\r\n\r\n* Python standard library's [`ast`](https://docs.python.org/3/library/ast.html#module-ast)\r\n* Python standard library's [`lib2to3`](https://docs.python.org/3/library/2to3.html?highlight=2to3#module-lib2to3)\r\n* Python standard library's [`2to3` tool](https://docs.python.org/3/library/2to3.html)\r\n* [jedi](https://github.com/davidhalter/jedi)\r\n* [rope](https://github.com/python-rope/rope/)\r\n* [pep8](https://github.com/PyCQA/pep8)\r\n* [astroid](https://github.com/landscapeio/astroid) and [pylint](http://www.pylint.org/)\r\n* [yapf](https://github.com/google/yapf)\r\n* [baron](https://github.com/Psycojoker/baron), [redbaron](https://github.com/Psycojoker/redbaron), and [pyfmt](https://github.com/Psycojoker/pyfmt)\r\n\r\nThe overlap between any of these projects is by no means complete, but the overlap between all of them is still significant, especially given they solve the same underlying problem of Python code representation. My hope here is we can examine whether the Python ecosystem really needs this many implementations of the Python syntax tree, and whether we can arrive at an ecosystem that has a unified base, with the diversity of effort occurring on more interesting problem spaces such as advanced refactoring and static analyses.\r\n\r\nI am hoping to start the discussion here in jedi's GitHub Issues because of the project's recent announcement of a [representation of Python code based upon an (improved?) fork of `lib2to3`](https://mail.python.org/pipermail/code-quality/2015-April/000534.html), which, from my understanding should provide a usable syntax tree representation for both Python 2 and Python 3.\r\n\r\nOne question is whether the jedi library's API is wide enough to support projects beyond jedi, and, if it's not, what additions must be made to make it more universal? We must acknowledge that each tool and library mentioned in the list above may only need some portion of information from the syntax tree, but that a good underlying representation would have the union of such information required by all client tools and libraries. Clearly some requirements gathering must be done, but my hope is there is a strong and reconcilable overlap in requirements between the projects.\r\n\r\nAnother question is, should this library be broken out into its own Python distribution package? This would allow the fundamental underlying library to move at its own pace independent of jedi's release schedule. This means jedi would become a client of library, as I hope other libraries and tools could.\r\n\r\nIt also seems that the base representation, in being an _accurate_ representation of Python code, may not be the most _convenient_ or appropriate interface through which a client should interact with the code, suggesting room for higher level APIs, provided by separate libraries that wrap the base representation, for example, how [pandas](http://pandas.pydata.org/) is to [numpy](http://www.numpy.org/). I think there exists the opportunity to create a library stack, in which the tools mentioned above become clients of the high-level libraries mentioned above, which in turn are clients of the base representation library. In fact, we can see discussion of these very ideas already within and between multiple projects, e.g., python-rope/rope#57, and Psycojoker/baron#61. I think now is the time to get serious about these efforts.\r\n\r\nFinally, I'd like to point to two inspirations that this collaboration of effort and unification can happen. The first inspiration is the scientific Python community's unification around the HDF5 Python stack, which I encourage you to read about in @scopatz's [excellent blog post about that effort](https://hdfgroup.org/wp/2015/09/python-hdf5-a-vision/). The second inspiration comes from @royrapoport's description of the way Netflix works in [his recent interview on Talk Python To Me](http://talkpython.fm/episodes/show/16/python-at-netflix). There, teams are permitted to make their own engineering decisions, but this inevitably creates duplication of effort. To counteract that duplication, Netflix periodically evaluates overlapping projects, and if their duplication is found technically unjustifiable, the project's maintainers must sit down with each other and figure out how to merge the code bases. While there's no fiduciary incentive in the case of the projects I've mentioned above here, there is still the incentive of reducing the amount of time, personally and collectively, spent solving very similar problems.\r\n\r\nI think a similar effort here to unify would be a great win for Python tooling and the Python community. My hope is this opens up the conversation between these various projects.\r\n\r\ncc @davidhalter @aligrudi @mcepl @Psycojoker @gwelymernans @hayd @jcrocholl @florentx @IanLee1521 @PCManticore\r\n\r\nI will also try to reach out to maintainers of other projects whom I could not find on GitHub, but if anyone here has contacts mentioned or omitted but interested, please reach out to them as well.\r\n\r\n@davidhalter, forgive me for hijacking your project's Issues for this.\r\n\r\nApologies also for anyone who is offended this discussion would happen on GitHub or in a GitHub issue. To me this seems the most open, publicly visible, conducive space to begin the conversation, though, as I suspect few people are members of the [Python code-quality mailing list](https://mail.python.org/mailman/listinfo/code-quality) when compared to the number of people impacted and interested."},"comment":{"url":"https://api.github.com/repos/davidhalter/jedi/issues/comments/230133016","html_url":"https://github.com/davidhalter/jedi/issues/630#issuecomment-230133016","issue_url":"https://api.github.com/repos/davidhalter/jedi/issues/630","id":230133016,"user":{"login":"Psycojoker","id":41827,"avatar_url":"https://avatars.githubusercontent.com/u/41827?v=3","gravatar_id":"","url":"https://api.github.com/users/Psycojoker","html_url":"https://github.com/Psycojoker","followers_url":"https://api.github.com/users/Psycojoker/followers","following_url":"https://api.github.com/users/Psycojoker/following{/other_user}","gists_url":"https://api.github.com/users/Psycojoker/gists{/gist_id}","starred_url":"https://api.github.com/users/Psycojoker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Psycojoker/subscriptions","organizations_url":"https://api.github.com/users/Psycojoker/orgs","repos_url":"https://api.github.com/users/Psycojoker/repos","events_url":"https://api.github.com/users/Psycojoker/events{/privacy}","received_events_url":"https://api.github.com/users/Psycojoker/received_events","type":"User","site_admin":false},"created_at":"2016-07-03T03:18:42Z","updated_at":"2016-07-03T03:18:42Z","body":"hello @Nurdok,\r\n\r\nYes (red)baron performances are not good mostly because I've been focusing my efforts on making it easy to do things that were (very) hard/way too annoying to do before with a nice API. I haven't done any work on performances and that's for later on my todo list (but I won't block contributions in that direction as long as they don't reduce too much code maintainability).\r\n\r\nFor your situation I would better look at either jedi or lib2to3 parser since they keep formatting information (but not the same way than (red)baron: here a token is responsible for the formatting behind him (or after, I have a doubt)) and are very fast. Be aware that lib2to3 is known to have some bugs: https://github.com/PyCQA/baron/issues/61#issuecomment-221965793 (and there is a bug about that around on some tool that use it but I can't find it anymore). I don't have information regarding the state of jedi.\r\n\r\nCheers,"}}
{"actor":{"display_login":"Psycojoker"}}