{"action":"created","issue":{"url":"https://api.github.com/repos/sschaef/tooling-research/issues/13","repository_url":"https://api.github.com/repos/sschaef/tooling-research","labels_url":"https://api.github.com/repos/sschaef/tooling-research/issues/13/labels{/name}","comments_url":"https://api.github.com/repos/sschaef/tooling-research/issues/13/comments","events_url":"https://api.github.com/repos/sschaef/tooling-research/issues/13/events","html_url":"https://github.com/sschaef/tooling-research/issues/13","id":134839757,"number":13,"title":"Semantic search engine","user":{"login":"sschaef","id":488530,"avatar_url":"https://avatars.githubusercontent.com/u/488530?v=3","gravatar_id":"","url":"https://api.github.com/users/sschaef","html_url":"https://github.com/sschaef","followers_url":"https://api.github.com/users/sschaef/followers","following_url":"https://api.github.com/users/sschaef/following{/other_user}","gists_url":"https://api.github.com/users/sschaef/gists{/gist_id}","starred_url":"https://api.github.com/users/sschaef/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sschaef/subscriptions","organizations_url":"https://api.github.com/users/sschaef/orgs","repos_url":"https://api.github.com/users/sschaef/repos","events_url":"https://api.github.com/users/sschaef/events{/privacy}","received_events_url":"https://api.github.com/users/sschaef/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-02-19T11:13:58Z","updated_at":"2016-02-19T17:55:29Z","closed_at":null,"body":"We need a semantic search engine for all of our tools. The key requirements are:\r\n\r\n- Index arbitrary semantic data\r\n- Provide a query API that allows users to search for the data they are interested in\r\n- Provide the search engine as a standalone library and as a web service \r\n- The index should be extensible in what can be indexed and how data can be classified with semantic information\r\n\r\nIn order to limit the scope of such a search engine, at the beginning we should focus at indexing concrete syntax trees (CST) for the programming language Scala. Indexed CSTs would already allow us to implement a lot of useful features including but not limited to:\r\n\r\n- Find references to arbitrary symbols (identifiers with semantic information)\r\n- Build a type hierarchy\r\n- Dependency hierarchy of projects\r\n- Show implementation classes of interfaces/traits/abstract/classes\r\n- Show super implementation of overridden symbol\r\n- Sort potential code completion matches\r\n\r\nA selection of projects that could need such a search engine:\r\n\r\n- [scala-ide](https://github.com/scala-ide/scala-ide) - The IDE already supports some of the features mentioned above but they all rely on the search engine that comes with Eclipse, which is not powerful enough to support all kind of use cases.\r\n- [scala-search](https://github.com/scala-ide/scala-search) - It was created to supersede the search engine that is shipped with scala-ide but it has severe limitations in so far that its goal is to only index trees. Also, the project is not standalone and therefore its indexer can't be used outside of Eclipse.\r\n- [intellij-scala](https://github.com/JetBrains/intellij-scala) - Similar to the scala-ide case but IntelliJ already has its own indexer for CSTs. Their indexer is still not powerful enough to index all kind of information but it is difficult to build such an indexer anyway. Since reusing the indexer of IntelliJ would limit us in prototyping a semantic search engine for everything, we shouldn't rely on it.\r\n- [scala-refactorings](https://github.com/scala-ide/scala-refactoring) - Refactorings right now depend on scala.reflect trees, which are ASTs and not CSTs. We could depend on scala.meta trees but the project is not there yet. Also, scala.meta provides only a CST for Scala and not an abstraction for all kind of programming languages or let alone for information that is not part of trees. The query API however would be an abstraction that could be used to search for the information that is needed by the refactoring implementations. And it would make some refactorings even programming language independent, since the query API abstracts over languages.\r\n- [ensime-server](https://github.com/ensime/ensime-server) - Similar cases as for the IDEs, the [Graphpocolypse](https://github.com/ensime/ensime-server/issues/1133) proposal shows that they also need something better.\r\n- [scaps](https://github.com/scala-search/scaps) - It relies heavily on indexing information that comes out of trees. Right now this project has two goals: Provide a search engine and classify the trees in a way that users can find what they need. The latter is more important for its users, the indexer is just there to speed the whole thing up. Each use case should be handled by a different project to make maintenance easier, where the indexer is what is described by this proposal and should be moved out of scaps.\r\n- [scala.meta](https://github.com/scalameta/scalameta) - For scala.meta it would be interesting to feed the indexer with its trees and to get them back again when they are needed. This would be mostly interesting for users who don't want to use the query API but who want to work directly on top of trees that are generated from a cache.\r\n\r\nThe goal of the semantic search project is to provide an abstraction that can be used by all of the tools mentioned above. It shall supersede all existing indexers used by any of the tools above and provide an API that makes it possible for these tools to rely on this single uber indexer.\r\n\r\nI have the chance to work several months full-time on this project and would like to explore the technical challenges that need to be solved in order to get such a uber indexer. Since I need to get something done the main goal is to first get the indexer up and running for the needs of various Scala tools and if further time remains to find out how to move it to the next level.\r\n\r\nI would like to get feedback by all maintainer of the tools mentioned above, namely @fommil @mlangc @misto @xeno-by @smarter @dragos @rorygraves @Luegg @wpopielarski"},"comment":{"url":"https://api.github.com/repos/sschaef/tooling-research/issues/comments/186331926","html_url":"https://github.com/sschaef/tooling-research/issues/13#issuecomment-186331926","issue_url":"https://api.github.com/repos/sschaef/tooling-research/issues/13","id":186331926,"user":{"login":"fommil","id":1914041,"avatar_url":"https://avatars.githubusercontent.com/u/1914041?v=3","gravatar_id":"","url":"https://api.github.com/users/fommil","html_url":"https://github.com/fommil","followers_url":"https://api.github.com/users/fommil/followers","following_url":"https://api.github.com/users/fommil/following{/other_user}","gists_url":"https://api.github.com/users/fommil/gists{/gist_id}","starred_url":"https://api.github.com/users/fommil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fommil/subscriptions","organizations_url":"https://api.github.com/users/fommil/orgs","repos_url":"https://api.github.com/users/fommil/repos","events_url":"https://api.github.com/users/fommil/events{/privacy}","received_events_url":"https://api.github.com/users/fommil/received_events","type":"User","site_admin":false},"created_at":"2016-02-19T17:55:29Z","updated_at":"2016-02-19T17:55:29Z","body":"Performance, memory size and extensibility. With relational databases, a table needs to be created for each feature, there is huge duplication of data (even just the extra keys). With a graph database, I'm hoping we can create the schema and write new features later."}}