{"action":"created","issue":{"url":"https://api.github.com/repos/gap-system/gap/issues/3089","repository_url":"https://api.github.com/repos/gap-system/gap","labels_url":"https://api.github.com/repos/gap-system/gap/issues/3089/labels{/name}","comments_url":"https://api.github.com/repos/gap-system/gap/issues/3089/comments","events_url":"https://api.github.com/repos/gap-system/gap/issues/3089/events","html_url":"https://github.com/gap-system/gap/issues/3089","id":387715480,"node_id":"MDU6SXNzdWUzODc3MTU0ODA=","number":3089,"title":"libgap API: Problems with GASMAN and StackBottom","user":{"login":"embray","id":676149,"node_id":"MDQ6VXNlcjY3NjE0OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/676149?v=4","gravatar_id":"","url":"https://api.github.com/users/embray","html_url":"https://github.com/embray","followers_url":"https://api.github.com/users/embray/followers","following_url":"https://api.github.com/users/embray/following{/other_user}","gists_url":"https://api.github.com/users/embray/gists{/gist_id}","starred_url":"https://api.github.com/users/embray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/embray/subscriptions","organizations_url":"https://api.github.com/users/embray/orgs","repos_url":"https://api.github.com/users/embray/repos","events_url":"https://api.github.com/users/embray/events{/privacy}","received_events_url":"https://api.github.com/users/embray/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-12-05T11:34:22Z","updated_at":"2018-12-06T16:44:32Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I have brought this issue up previously on the Sage Trac, but this is a generic problem with embedding libgap in larger applications, and it is a multi-faceted problem which I thought should be clearly summarized somewhere.\r\n\r\nThe core of the problem is that the GAP kernel, as designed, assumes that all code which uses GAP objects (tracked through the GASMAN GC), is going to run in a context with either a call to `InitializeGap` as a parent stack frame, or at least have a common parent with the call to `InitializeGap`.  This is illustrated in the fact that `InitializeGap` uses a pointer to some main function's `argc` variable to set the `StackBottom` variable in the GC (this itself is a problem which I'll list later).\r\n\r\nThis assumption for setting `StackBottom` is safe for GAP itself, but not for embedding in another application with libgap.  Consider the following scenario, where `GAP_Initialize` is called by some program embedding GAP, where GAP initialization is delayed until necessary, and possibly not called until deep into some other function call (forgive my ASCII art):\r\n\r\n```\r\n#0 main()\r\n#1  |-<some func>\r\n#2      |- <some func 2>\r\n             .\r\n             .\r\n#42           |- <some func 3>   // 42 stack frames deep (or any arbitrary large N)\r\n#43                |- GAP_Initialize()  // We need to use GAP now for some computation\r\n                                         // StackBottom is set *here*\r\n```\r\n\r\nNow imagine that later in the application there is some other function called which uses GAP, but less deeply nested than the one that happened to call `GAP_Initialize`; it also happens to result in some nested GAP function calls:\r\n\r\n```\r\n#0 main()\r\n#1  |-<some func>\r\n#2      |- <some func 2>\r\n             .\r\n             .\r\n#10           |- <some func 3>\r\n#11                |- GAP_EvalString(....)\r\n                       .\r\n                       .\r\n#22                    |- CALL_NARGS(....)  // Some objects are allocated on the stack here\r\n#23                         |- CALL_NARGS(...)  // \"top\" of stack is here; garbage collection occurs\r\n.  \r\n.\r\n.\r\n#43 (old)          // StackBottom is still down here somewhere...\r\n```\r\n\r\nNow we have (on x86 architectures) `StackTop > StackBottom`.  `GenStackFuncBags` does handle the case where  `top > bottom`, but that \"seems\" to be more in order to handle different architectures where the stack grows in different directions, than some deliberate attempt to handle this case.  Because now there are stack frames possibly containing young bags that haven't been swept yet which are above the current \"top\" of the stack. \r\n\r\nSo in the illustration above, `GenStackFuncBags` will sweep from `#43`, up through a bunch of garbage, dead stack frames, until `#23`, and will stop there without continuing upward to `#22` and so on which may still contain some unmarked bags in local variables which will then be marked dead despite being still very much in use.\r\n\r\nThis seems to be a problem to me, but is there something obvious I'm missing in this analysis.\r\n\r\n@sebasguts , is this problem handled in any way by GAPJulia?\r\n\r\n\r\nAs far as I can tell, if the above analysis is a real problem, then that places the onus on any application which embeds GAP to *ensure* that GAP is initialized very early on in the application so that anything else that uses GAP and creates GAP objects happens in a \"deeper\" stack frame.\r\n\r\nFor a custom application this is less of a problem as one has more control over when `GAP_Initialize` is actually called (though it is still a problem if one wants to avoid the overhead of `GAP_Initialize` by calling it lazily only when needed).\r\n\r\nFor embedding in library for another language like Python, however, it's a bigger problem.  Because in writing a Python module that wraps GAP we cannot easily guarantee when that module will be imported relative to execution of the rest of the application (much less relative to Python's `main()` function).\r\n\r\nThere are multiple possible solutions to this problem and I'm not sure which is the best one.  If nothing else it ought to be possible, when calling `GAP_Initialize`, to provide a custom value for what address should be set to `StackBottom`, rather than just assuming it's where `GAP_Initialize` is called from.  Additionally I don't think it should double as a pointer to the `argc` value passed to `InitializeGap`, as that then places the undue burden on the client code of ensuring that whatever we pass as `StackBottom` points to the `argc` argument we want to pass to GAP.\r\n\r\nEven then, however, it's not always obvious from the perspective of client code where an appropriate address is to set `StackBottom` to--this is a rather low-level concern for most users of the code.  ISTM the best option is to provide some `GAP_Enter` and `GAP_Exit` gates that should surround any client code that calls libgap functions, and that possibly has GAP objects on the stack.  `GAP_Enter` would mark the location of `StackBottom` from where it is called (it would likely be a macro), and would include protection for recursion (hence the need for a `GAP_Exit` as well).\r\n\r\nI'm not sure that there's really a better way, unless there were some way this could be managed automatically by the API-level functions, but my experience with GAP's GC is not at a level where I can see a better way."},"comment":{"url":"https://api.github.com/repos/gap-system/gap/issues/comments/444941060","html_url":"https://github.com/gap-system/gap/issues/3089#issuecomment-444941060","issue_url":"https://api.github.com/repos/gap-system/gap/issues/3089","id":444941060,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NDk0MTA2MA==","user":{"login":"embray","id":676149,"node_id":"MDQ6VXNlcjY3NjE0OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/676149?v=4","gravatar_id":"","url":"https://api.github.com/users/embray","html_url":"https://github.com/embray","followers_url":"https://api.github.com/users/embray/followers","following_url":"https://api.github.com/users/embray/following{/other_user}","gists_url":"https://api.github.com/users/embray/gists{/gist_id}","starred_url":"https://api.github.com/users/embray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/embray/subscriptions","organizations_url":"https://api.github.com/users/embray/orgs","repos_url":"https://api.github.com/users/embray/repos","events_url":"https://api.github.com/users/embray/events{/privacy}","received_events_url":"https://api.github.com/users/embray/received_events","type":"User","site_admin":false},"created_at":"2018-12-06T16:44:32Z","updated_at":"2018-12-06T16:44:32Z","author_association":"CONTRIBUTOR","body":"See #3096 for my (thus far quite successful, at least for practical purposes) prototype of a solution."}}
{"actor":{"display_login":"embray"}}