{"action":"created","issue":{"url":"https://api.github.com/repos/racket/racket/issues/2154","repository_url":"https://api.github.com/repos/racket/racket","labels_url":"https://api.github.com/repos/racket/racket/issues/2154/labels{/name}","comments_url":"https://api.github.com/repos/racket/racket/issues/2154/comments","events_url":"https://api.github.com/repos/racket/racket/issues/2154/events","html_url":"https://github.com/racket/racket/issues/2154","id":336729454,"node_id":"MDU6SXNzdWUzMzY3Mjk0NTQ=","number":2154,"title":"Reconsider `#:extend-stop-ids?` argument to `local-expand`","user":{"login":"michaelballantyne","id":1429331,"node_id":"MDQ6VXNlcjE0MjkzMzE=","avatar_url":"https://avatars3.githubusercontent.com/u/1429331?v=4","gravatar_id":"","url":"https://api.github.com/users/michaelballantyne","html_url":"https://github.com/michaelballantyne","followers_url":"https://api.github.com/users/michaelballantyne/followers","following_url":"https://api.github.com/users/michaelballantyne/following{/other_user}","gists_url":"https://api.github.com/users/michaelballantyne/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelballantyne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelballantyne/subscriptions","organizations_url":"https://api.github.com/users/michaelballantyne/orgs","repos_url":"https://api.github.com/users/michaelballantyne/repos","events_url":"https://api.github.com/users/michaelballantyne/events{/privacy}","received_events_url":"https://api.github.com/users/michaelballantyne/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-06-28T18:23:52Z","updated_at":"2018-06-28T19:29:37Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I started a discussion with @lexi-lambda on slack about the `#:extend-stop-ids?` addition to `local-expand`. She suggested I include others who had expressed an interest, so here it is.\r\n\r\nI'm not convinced that uses of the feature can be reliable, so I'm proposing it be removed before the Racket 7 release.\r\n\r\nMy concern is that any macro that needs to use the feature will be fragile. The previous rule of adding all the core forms to the stop list for any partial expansion had the effect of ensuring that when a macro is expanded in expression position, it will always be a full rather than partial expansion.\r\n\r\n`let-syntax`, `syntax-parameterize`, and anything using first-class definition contexts either rely on this guarantee, or break partial expansion of expressions by their own use of local-expand. (syntax-parameterize could be implemented to work either way). If a let-syntax expression is partially expanded such that there remain unexpanded uses of bound macro, later re-expansion will fail with an identifier out of context error.\r\n\r\nIn Alexis's use-case in Hackett, she has to be careful to not compile Hackett forms to Racket expressions that use these features, and had trouble with `match` using `syntax-parameterize`. Thus `#:extend-stop-ids?` works given you can assume a closed world of macros you might expand, and as long as the implementation of those macros isn't changed in the future to use local macro binding or syntax parameters in a way that binders and references could be separated by one of the forms expansion stops at. Other cases will break.\r\n\r\nIf a language like Hackett doesn't follow the same protocol as Racket for its macros, it would be better to define a custom expander (as Alexis did for Hackett's type language here: https://lexi-lambda.github.io/), rather than re-use Racket's forms and macros. The core forms of that language and macros that expand to them know to expect the different protocol. In the case of Hackett's partial expansion, these forms would be careful to not rely on local syntax bindings that their re-expansion doesn't re-create. For example, a typed \"let\" core form could use a syntax binding to attach type information to any references, but rather than expand away as in Turnstile, the form would expand to itself. Subsequent re-expansions would re-establish the syntax binding for the type information just as re-expansions of `let` re-establish the normal let binding. A toy sample of such an expander is here: https://gist.github.com/michaelballantyne/d040e4cdec8a3484fb49adaa0ce5a227\r\n\r\nThe implementation of syntax parameters should probably expose something that would allow languages like Hackett to define their own syntax-parameterize form, compatible with their expansion protocol.\r\n\r\nI'm curious about other people's interest in `#:extend-stop-ids?`. Do you have use-cases that you think would be robust?\r\n\r\n@florence @samth @mflatt "},"comment":{"url":"https://api.github.com/repos/racket/racket/issues/comments/401146948","html_url":"https://github.com/racket/racket/issues/2154#issuecomment-401146948","issue_url":"https://api.github.com/repos/racket/racket/issues/2154","id":401146948,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMTE0Njk0OA==","user":{"login":"lexi-lambda","id":759911,"node_id":"MDQ6VXNlcjc1OTkxMQ==","avatar_url":"https://avatars0.githubusercontent.com/u/759911?v=4","gravatar_id":"","url":"https://api.github.com/users/lexi-lambda","html_url":"https://github.com/lexi-lambda","followers_url":"https://api.github.com/users/lexi-lambda/followers","following_url":"https://api.github.com/users/lexi-lambda/following{/other_user}","gists_url":"https://api.github.com/users/lexi-lambda/gists{/gist_id}","starred_url":"https://api.github.com/users/lexi-lambda/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lexi-lambda/subscriptions","organizations_url":"https://api.github.com/users/lexi-lambda/orgs","repos_url":"https://api.github.com/users/lexi-lambda/repos","events_url":"https://api.github.com/users/lexi-lambda/events{/privacy}","received_events_url":"https://api.github.com/users/lexi-lambda/received_events","type":"User","site_admin":false},"created_at":"2018-06-28T19:29:37Z","updated_at":"2018-06-28T19:29:37Z","author_association":"MEMBER","body":"I’ve already spoken to Michael about this a little bit, but let me offer a little extra context from my perspective.\r\n\r\nThe main thing I want to say is that the current behavior of the stop list *is*, at least with the current expander, arbitrary. Let’s look at which forms are implicitly added to the stop list: `begin`, `quote`, `set!`, `#%plain-lambda`, `case-lambda`, `let-values`, `letrec-values`, `if`, `begin0`, `with-continuation-mark`, `letrec-syntaxes+values`, `#%plain-app`, `#%expression`, `#%top`, and `#%variable-reference`. Some observations:\r\n\r\n  1. First of all, *this list is **not** the list of Racket’s core forms*. In fact, it’s not even a subset of the forms that appear in fully-expanded programs: `#%expression` is a core form, but it doesn’t appear in fully-expanded code. A variety of core forms are *not* included in the stop list, like `define-values`/`define-syntaxes`, `begin-for-syntax`, `quote-syntax`, and `module`/`module*`(!). Indeed, everything implicitly added to the stop list can appear in expression position. This choice is definitely not intrinsic to the operation of the expander.\r\n\r\n  2. A couple of these forms don’t really do anything at all, so adding them to the stop list is largely pointless. Things like `quote` and `#%variable-reference` are the most obvious offenders. Technically, adding them to the stop list does do *something*—it means malformed uses of `quote` and `#%variable-reference` (like writing `(quote)` or `(quote 1 2)`) won’t be reported as syntax errors, but it definitely doesn’t impact the partial expansion protocol Michael mentions.\r\n\r\n     Similar but slightly different is `#%top`. Note that this refers to *the* `#%top` from `'#%kernel`, not any implicit introduction of `#%top`. Expanding this in a module will only ever report an unbound identifier error, and expanding it at the top level won’t do anything.\r\n\r\n  3. Many other forms still aren’t binding forms, so they can likewise be safely expanded without losing definitions. This includes `begin`, `set!`, `if`, `begin0`, `with-continuation-mark`, `#%plain-app`, and `#%expression`. There is no technical reason to implicitly add these to the stop list.\r\n\r\nThese points imply that the implicit stop list could *at least* be simplified to the binding forms: `#%plain-lambda`, `case-lambda`, `let-values`, `letrec-values`, and `letrec-syntaxes+values`. Partially expanding these can sometimes be unsafe. This occurs most obviously with `letrec-syntaxes+values`, with which we can trivially construct a bad program:\r\n\r\n```racket\r\n(define-syntax-rule (stop x) x)\r\n\r\n(let-syntax ([x (make-rename-transformer #'void)])\r\n  (stop (x)))\r\n```\r\n\r\nIf we partially expand that program, stopping at `stop`, then expand the partially-expanded result, we get an out-of-context error. The problem is that `let-syntax` effectively disappears after expansion, so we end up with this partially-expanded fragment:\r\n\r\n```racket\r\n(let-values ()\r\n  (stop (x)))\r\n```\r\n\r\nNow, the remaining `x` identifier has the right scopes to refer to the original syntax binding, but the value of the binding is gone, and we get an out of context error.\r\n\r\nRuntime binding forms are more subtle. To a first approximation, partially expanding them is actually safe. If we reconstruct the same program using `let` instead of `let-syntax`, it would work just fine:\r\n\r\n```racket\r\n(let ([x void])\r\n  (stop (x)))\r\n```\r\n\r\nAfter partial expansion, the program fragment would be largely unchanged. The only difference would be the creation of an additional scope. This works out okay, since when we go to continue expansion, the `x` is still bound. However, this is possible to break with a more devious program. If we expand a macro during partial expansion that stashes identifiers in some piece of compile-time state, we can still end up with binding inconsistency, since the stashed identifier will not be enriched by subsequently added local binding scopes.\r\n\r\nTherefore, I make the case that *even if* we decide that `#:extend-stop-ids?` is too dangerous a hammer, we can at least provide an option that simplifies the stop list to exclusively binding forms. But here’s the catch: **we don’t actually know which forms are binding forms**, due to the existence of first-class definition contexts. This means that I can write my own version of `let-syntax` that never even expands to Racket’s `letrec-syntaxes+values`, since it can just use first-class definition contexts alone. Even the previous behavior of the stop list can’t prevent this, and partial expansion through such a form breaks expansion in the same way.\r\n\r\nOn the other hand, one can argue that including the implicit addition of items to the stop list at least provides an implicit protocol to allow macro authors to write macros that are safe in this regard. Before using first-class definition contexts, they can wrap their expansion in a (possibly-empty) local binding form, forcing any partial expansion to stop there. This protocol is pretty awkward, and my guess is that almost nobody would even *begin* to consider it when writing macros using first-class definition contexts (indeed, I’d be curious to know if, say, @rmculpepper was even aware of this implicit protocol’s existence), but at least the potential protocol exists.\r\n\r\nTherefore, I’d reframe this discussion slightly. I think the debate is not whether or not `#:extend-stop-ids?` should go away completely, but whether it should be weakened somehow to a version that only stops on binding forms. I think (correct me, Michael, if I’m wrong) that captures the essence of Michael’s discomfort with the change."}}
{"actor":{"display_login":"lexi-lambda"}}