{"action":"created","issue":{"url":"https://api.github.com/repos/google/kythe/issues/1960","repository_url":"https://api.github.com/repos/google/kythe","labels_url":"https://api.github.com/repos/google/kythe/issues/1960/labels{/name}","comments_url":"https://api.github.com/repos/google/kythe/issues/1960/comments","events_url":"https://api.github.com/repos/google/kythe/issues/1960/events","html_url":"https://github.com/google/kythe/issues/1960","id":335626658,"node_id":"MDU6SXNzdWUzMzU2MjY2NTg=","number":1960,"title":"Proposal: Per-language \"key compilations\" for language intrinsics [phab:T242]","user":{"login":"kythe","id":10677145,"node_id":"MDQ6VXNlcjEwNjc3MTQ1","avatar_url":"https://avatars2.githubusercontent.com/u/10677145?v=4","gravatar_id":"","url":"https://api.github.com/users/kythe","html_url":"https://github.com/kythe","followers_url":"https://api.github.com/users/kythe/followers","following_url":"https://api.github.com/users/kythe/following{/other_user}","gists_url":"https://api.github.com/users/kythe/gists{/gist_id}","starred_url":"https://api.github.com/users/kythe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kythe/subscriptions","organizations_url":"https://api.github.com/users/kythe/orgs","repos_url":"https://api.github.com/users/kythe/repos","events_url":"https://api.github.com/users/kythe/events{/privacy}","received_events_url":"https://api.github.com/users/kythe/received_events","type":"User","site_admin":false},"labels":[{"id":932810970,"node_id":"MDU6TGFiZWw5MzI4MTA5NzA=","url":"https://api.github.com/repos/google/kythe/labels/P2","name":"P2","color":"d97c5c","default":false},{"id":932815123,"node_id":"MDU6TGFiZWw5MzI4MTUxMjM=","url":"https://api.github.com/repos/google/kythe/labels/indexing","name":"indexing","color":"f9d66b","default":false},{"id":975094874,"node_id":"MDU6TGFiZWw5NzUwOTQ4NzQ=","url":"https://api.github.com/repos/google/kythe/labels/phabricator%20task","name":"phabricator task","color":"efd57f","default":false}],"state":"open","locked":false,"assignee":{"login":"creachadair","id":19258,"node_id":"MDQ6VXNlcjE5MjU4","avatar_url":"https://avatars1.githubusercontent.com/u/19258?v=4","gravatar_id":"","url":"https://api.github.com/users/creachadair","html_url":"https://github.com/creachadair","followers_url":"https://api.github.com/users/creachadair/followers","following_url":"https://api.github.com/users/creachadair/following{/other_user}","gists_url":"https://api.github.com/users/creachadair/gists{/gist_id}","starred_url":"https://api.github.com/users/creachadair/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/creachadair/subscriptions","organizations_url":"https://api.github.com/users/creachadair/orgs","repos_url":"https://api.github.com/users/creachadair/repos","events_url":"https://api.github.com/users/creachadair/events{/privacy}","received_events_url":"https://api.github.com/users/creachadair/received_events","type":"User","site_admin":false},"assignees":[{"login":"creachadair","id":19258,"node_id":"MDQ6VXNlcjE5MjU4","avatar_url":"https://avatars1.githubusercontent.com/u/19258?v=4","gravatar_id":"","url":"https://api.github.com/users/creachadair","html_url":"https://github.com/creachadair","followers_url":"https://api.github.com/users/creachadair/followers","following_url":"https://api.github.com/users/creachadair/following{/other_user}","gists_url":"https://api.github.com/users/creachadair/gists{/gist_id}","starred_url":"https://api.github.com/users/creachadair/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/creachadair/subscriptions","organizations_url":"https://api.github.com/users/creachadair/orgs","repos_url":"https://api.github.com/users/creachadair/repos","events_url":"https://api.github.com/users/creachadair/events{/privacy}","received_events_url":"https://api.github.com/users/creachadair/received_events","type":"User","site_admin":false}],"milestone":null,"comments":6,"created_at":"2018-06-26T01:40:41Z","updated_at":"2018-06-27T21:29:27Z","closed_at":null,"author_association":"COLLABORATOR","body":"_Created by **fromberger** at 2017-04-19 06:49:45_: \n\nAside from pure lambda calculus, most programming languages come equipped with \"intrinsics\", which for discussion I'll define as the collection of entities (types, functions, variables, properties, etc.) that are built-in or //intrinsic// to the language. Examples include Go's built-in `append`, function, core Python types like `str` and `unicode` that are (essentially) hardwired into `__builtin__`, built-in types like `int`, `float`, and `char` in C and C-derived languages,[1] and so forth. More broadly, this terminology may be slightly abused to include code that is technically library code (in that it may be separately importable), but which is usually not seen in source format because it is precompiled in a well-known location. (Many of the JVM built-in types like `java.lang.String` are of this character, and likewise the Go standard library as consumed by the `go build` command).\n\nFor indexing, intrinsics present a tricky challenge: You'd like to index them, since they are an important part of the language. But the nature of built-in objects is that they do not present as \"compilations\" in the usual sense: You don't necessarily get source for them as part of the build. \n\nIn some cases you could argue that cross-references for language intrinsics aren't necessary that useful: It isn't necessarily that helpful for a code browser to show you all the places where the built-in `bool` type is used, for example.[2] But even if you choose not to emit cross-references for intrinsics, tools need to be able to find links to things like canonical documentation (e.g., to have uses of `str` link to http://docs.python.org/3/library/stdtypes.html#str). This becomes especially important for \"hosted\" language such as Python, JavaScript, or Lua, where there are often additional intrinsics provided by the host environment (e.g., the browser, for ECMAScript applications) that aren't necessarily visible to the programmer given only the language specification.\n\nWe have traditionally handled this by having the language indexer generate data for intrinsics opportunistically: The first time the indexer sees a usage of some intrinsic object, it generates the relevant graph artifacts for it and then sets a flag to remember that it did so, so that it won't repeat the effort (or the data). This works reasonably well for small corpora, but has some pernicious side-effects:\n\n1. The indexer is no longer hermetic, since the cache carries state across multiple compilations.\n2. (For that reason) Indexer output can't be safely cached, since outputs may vary depending on order of analysis.\n3. On a large corpus, there may be a lot of duplicate data attributable to intrinsics, which increases storage and processing costs (sometimes quite substantially).\n\nBecause this issue affects virtually all languages in a similar way, I'd like to propose a more hermetic and reliable solution.\n\n**Proposal:** For any language //X//, define a **key compilation** for //X// as a compilation unit with no source files or required inputs, whose VName has the following structure:\n\n\n```\n{\n   \"language\":  \"X\",\n   \"corpus\":    \"kythe/intrinsics\",\n   \"path\":      \"\",\n   \"root\":      \"\"\n}\n```\n\nIn other words the compilation has the language label for //X//, the special corpus label `kythe/intrinsics`, and all other VName fields empty except (optionally) the signature, which //may// contain a language-version label (e.g., `java8`, `python3`, etc.). The intended use of such a compilation is that:\n\n1. When indexing, generate exactly one such key compilation unit for each language to be indexed.\n2. Add the key compilation to the work list for the indexers of that language.\n3. Teach the indexers to recognize the key compilation, and in response to generate whatever data are useful for the intrinsics of the language (once).\n\nThis approach has the benefit that you need only generate the data for intrinsics once per corpus, and moreover gives you the ability to index intrinsics entirely separately and cache their results. Moreover, given that intrinsics change only rarely, you can sensibly afford to cache these data for a long time—it may only be necessary to update them when the indexer binary changes, or when a new release of the language is encountered.\n\nImplementing this proposal fully will require that each language indexer be taught to recognize key compilations for its primary language label, and that any indexing installation be instrumented to inject a key compilation for each language sometime during the process. Otherwise, however, this should not require any schema changes (unless otherwise needed to model some specific language's intrinsics).\n\n**Notes:**\n\n[1] Actually `int` is a somewhat subtle case. A C compiler may define (e.g.) `int` via a typedef in a header file, mapped to some \"real\" (concrete) type with a different name; so the concrete type may vary in size and signedness depending on platform, flags, etc.\n[2] It might be fun curiosity, but probably not that productive. And to answer such a query quickly may impose a substantial storage footprint."},"comment":{"url":"https://api.github.com/repos/google/kythe/issues/comments/400835103","html_url":"https://github.com/google/kythe/issues/1960#issuecomment-400835103","issue_url":"https://api.github.com/repos/google/kythe/issues/1960","id":400835103,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMDgzNTEwMw==","user":{"login":"kythe","id":10677145,"node_id":"MDQ6VXNlcjEwNjc3MTQ1","avatar_url":"https://avatars2.githubusercontent.com/u/10677145?v=4","gravatar_id":"","url":"https://api.github.com/users/kythe","html_url":"https://github.com/kythe","followers_url":"https://api.github.com/users/kythe/followers","following_url":"https://api.github.com/users/kythe/following{/other_user}","gists_url":"https://api.github.com/users/kythe/gists{/gist_id}","starred_url":"https://api.github.com/users/kythe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kythe/subscriptions","organizations_url":"https://api.github.com/users/kythe/orgs","repos_url":"https://api.github.com/users/kythe/repos","events_url":"https://api.github.com/users/kythe/events{/privacy}","received_events_url":"https://api.github.com/users/kythe/received_events","type":"User","site_admin":false},"created_at":"2018-06-27T21:29:27Z","updated_at":"2018-06-27T21:29:27Z","author_association":"COLLABORATOR","body":"_**fromberger** wrote at 2017-04-28 19:39:56_:\n\nI think it depends what you mean by \"unavailable\".\n\nIf the source is not available while you're indexing a target that depends on it, that shouldn't matter. That is in fact the //preferred// model of indexing with Kythe, which is intended to allow each compilation to be indexed in isolation.\n\nIf the source is //never// available, then it would make sense to treat that library as an intrinsic. If, however, it's simply that the source lives in another repo, and you need to index it separately and combine the data later, that's not an important use-case for intrinsics."}}
{"actor":{"display_login":"kythe"}}