{"action":"created","issue":{"url":"https://api.github.com/repos/JanitorTechnology/janitor/issues/249","repository_url":"https://api.github.com/repos/JanitorTechnology/janitor","labels_url":"https://api.github.com/repos/JanitorTechnology/janitor/issues/249/labels{/name}","comments_url":"https://api.github.com/repos/JanitorTechnology/janitor/issues/249/comments","events_url":"https://api.github.com/repos/JanitorTechnology/janitor/issues/249/events","html_url":"https://github.com/JanitorTechnology/janitor/issues/249","id":291880495,"number":249,"title":"Unstable IDE service in one container","user":{"login":"jankeromnes","id":599268,"avatar_url":"https://avatars2.githubusercontent.com/u/599268?v=4","gravatar_id":"","url":"https://api.github.com/users/jankeromnes","html_url":"https://github.com/jankeromnes","followers_url":"https://api.github.com/users/jankeromnes/followers","following_url":"https://api.github.com/users/jankeromnes/following{/other_user}","gists_url":"https://api.github.com/users/jankeromnes/gists{/gist_id}","starred_url":"https://api.github.com/users/jankeromnes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jankeromnes/subscriptions","organizations_url":"https://api.github.com/users/jankeromnes/orgs","repos_url":"https://api.github.com/users/jankeromnes/repos","events_url":"https://api.github.com/users/jankeromnes/events{/privacy}","received_events_url":"https://api.github.com/users/jankeromnes/received_events","type":"User","site_admin":false},"labels":[{"id":565317726,"url":"https://api.github.com/repos/JanitorTechnology/janitor/labels/Bug","name":"Bug","color":"eb0000","default":false},{"id":481421878,"url":"https://api.github.com/repos/JanitorTechnology/janitor/labels/IDE","name":"IDE","color":"e10075","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-01-26T11:56:17Z","updated_at":"2018-05-28T15:16:55Z","closed_at":"2018-01-30T12:59:05Z","author_association":"MEMBER","body":"The Cloud9 IDE in a brand new Servo container on OVH1 started misbehaving today (thanks a lot to @birtles for [reporting it](https://twitter.com/brianskold/status/956808718435274752) and helping troubleshoot it!)\r\n\r\nThe IDE was basically unusable, Editor and Terminal not working properly, and it was displaying errors like:\r\n\r\n```\r\nFailed to write to 'state.settings'. EDISCONNECT: Agent disconnected .Reconnecting...\r\n```\r\n\r\nand\r\n\r\n```\r\nLanguage server could not be loaded; some language features have been disabled\r\n```\r\n\r\nOpening the DevTools revealed that many requests were getting a `499` error status (on VFS URLs like https://ovh1.janitor.technology/vfs/1/9cbOePWuii5bALJ2?access_token=token).\r\n\r\nInspecting the Cloud9 SDK error log with `less /tmp/c9sdk-stderr---supervisor-ejOQPD.log` revealed contiuous VFS errors, likely to be the cause of this malfunction:\r\n\r\n```\r\nError: 'nak' utility failed with exit code 1 and stderr: 'path.js:424\r\n    var path = (i >= 0) ? arguments[i] : process.cwd();\r\n                                                 ^\r\n\r\n    var path = (i >= 0) ? arguments[i] : process.cwd();\r\n                                                 ^\r\n\r\nError: ENOENT: no such file or directory, uv_cwd\r\n    at Error (native)\r\n    at Object.posix.resolve (path.js:424:50)\r\n    at Function.Module._resolveLookupPaths (module.js:249:17)\r\n    at Function.Module._resolveFilename (module.js:316:31)\r\n    at Function.Module._load (module.js:276:25)\r\n    at Module.require (module.js:353:17)\r\n    at require (internal/module.js:12:17)\r\n    at Object.<anonymous> (/home/user/.c9sdk/node_modules/nak/bin/nak:2:1)\r\n    at Module._compile (module.js:409:26)\r\n    at Object.Module._extensions..js (module.js:416:10)\r\n'\r\n    at EventEmitter.<anonymous> (/home/user/.c9sdk/plugins/c9.vfs.server/filelist.js:99:31)\r\n    at emitTwo (events.js:126:13)\r\n    at EventEmitter.emit (events.js:214:7)\r\n    at Parent.onExit (/home/user/.c9sdk/node_modules/vfs-socket/consumer.js:280:17)\r\n1:8)\r\n    at emitOne (events.js:116:13)\r\n    at Transport.emit (events.js:211:7)\r\n    at /home/user/.c9sdk/node_modules/smith/smith.js:96:14\r\n    at parse (/home/user/.c9sdk/node_modules/smith/smith.js:252:21)\r\n    at Socket.onData (/home/user/.c9sdk/node_modules/smith/smith.js:101:9)\r\nError: 'nak' utility failed with exit code 1 and stderr: 'path.js:424\r\n    var path = (i >= 0) ? arguments[i] : process.cwd();\r\n                                                 ^\r\nError: ENOENT: no such file or directory, uv_cwd\r\n    at Error (native)\r\n    at Object.posix.resolve (path.js:424:50)\r\n    at Function.Module._resolveLookupPaths (module.js:249:17)\r\n    at Function.Module._resolveFilename (module.js:316:31)\r\n    at Function.Module._load (module.js:276:25)\r\n    at Module.require (module.js:353:17)\r\n    at require (internal/module.js:12:17)\r\n    at Object.<anonymous> (/home/user/.c9sdk/node_modules/nak/bin/nak:2:1)\r\n    at Module._compile (module.js:409:26)\r\n    at Object.Module._extensions..js (module.js:416:10)\r\n'\r\nt.js:99:31)\r\n    at emitTwo (events.js:126:13)\r\n    at EventEmitter.emit (events.js:214:7)\r\n17)\r\n1:8)\r\n    at emitOne (events.js:116:13)\r\n    at Transport.emit (events.js:211:7)\r\n    at /home/user/.c9sdk/node_modules/smith/smith.js:96:14\r\n    at parse (/home/user/.c9sdk/node_modules/smith/smith.js:252:21)\r\n    at Socket.onData (/home/user/.c9sdk/node_modules/smith/smith.js:101:9)\r\nVFS Exception in function 'registerHandlers':\r\n Error: ENOENT: no such file or directory, uv_cwd\r\n    at Error (native)\r\n    at Object.posix.resolve (path.js:424:50)\r\n    at Function.Module._nodeModulePaths (module.js:190:15)\r\n    at createRequire (jsonalyzer_server:275:33)\r\n    at loadPlugin (jsonalyzer_server:253:19)\r\n    at registerHandler (jsonalyzer_server:131:5)\r\n    at async.forEachSeries.callback.summaries (jsonalyzer_server:107:13)\r\n    at iterate (jsonalyzer_server:328:13)\r\n    at Object.async.forEachSeries (jsonalyzer_server:344:9)\r\n    at Object.registerHandlers (jsonalyzer_server:104:11)\r\nScheduling process exit\r\nVFS Exception in function 'registerHandlers':\r\n Error: ENOENT: no such file or directory, uv_cwd\r\n    at Error (native)\r\n    at Object.posix.resolve (path.js:424:50)\r\n    at Function.Module._nodeModulePaths (module.js:190:15)\r\n    at createRequire (jsonalyzer_server:275:33)\r\n    at loadPlugin (jsonalyzer_server:253:19)\r\n    at registerHandler (jsonalyzer_server:131:5)\r\n    at async.forEachSeries.callback.summaries (jsonalyzer_server:107:13)\r\n    at iterate (jsonalyzer_server:328:13)\r\n    at Object.async.forEachSeries (jsonalyzer_server:344:9)\r\n    at Object.registerHandlers (jsonalyzer_server:104:11)\r\nScheduling process exit\r\nVFS Exception in function 'registerHandlers':\r\n Error: ENOENT: no such file or directory, uv_cwd\r\n    at Error (native)\r\n    at Object.posix.resolve (path.js:424:50)\r\n    at Function.Module._nodeModulePaths (module.js:190:15)\r\n    at createRequire (jsonalyzer_server:275:33)\r\n    at loadPlugin (jsonalyzer_server:253:19)\r\n    at registerHandler (jsonalyzer_server:131:5)\r\n    at async.forEachSeries.callback.summaries (jsonalyzer_server:107:13)\r\n    at iterate (jsonalyzer_server:328:13)\r\n    at Object.async.forEachSeries (jsonalyzer_server:344:9)\r\n    at Object.registerHandlers (jsonalyzer_server:104:11)\r\nScheduling process exit\r\n[...]\r\n```\r\n\r\nBy searching around, this issue looks similar to https://github.com/npm/npm/issues/10983 \r\n\r\nWe were able to work around this issue by simply creating a new container, which worked fine. Hopefully this was just a temporary bug, associated with that particular Cloud9 SDK version, and we won't see again.\r\n\r\nBut if you do see this bug again, please comment on this issue."},"comment":{"url":"https://api.github.com/repos/JanitorTechnology/janitor/issues/comments/392552634","html_url":"https://github.com/JanitorTechnology/janitor/issues/249#issuecomment-392552634","issue_url":"https://api.github.com/repos/JanitorTechnology/janitor/issues/249","id":392552634,"user":{"login":"jankeromnes","id":599268,"avatar_url":"https://avatars2.githubusercontent.com/u/599268?v=4","gravatar_id":"","url":"https://api.github.com/users/jankeromnes","html_url":"https://github.com/jankeromnes","followers_url":"https://api.github.com/users/jankeromnes/followers","following_url":"https://api.github.com/users/jankeromnes/following{/other_user}","gists_url":"https://api.github.com/users/jankeromnes/gists{/gist_id}","starred_url":"https://api.github.com/users/jankeromnes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jankeromnes/subscriptions","organizations_url":"https://api.github.com/users/jankeromnes/orgs","repos_url":"https://api.github.com/users/jankeromnes/repos","events_url":"https://api.github.com/users/jankeromnes/events{/privacy}","received_events_url":"https://api.github.com/users/jankeromnes/received_events","type":"User","site_admin":false},"created_at":"2018-05-28T15:16:55Z","updated_at":"2018-05-28T15:16:55Z","author_association":"MEMBER","body":"Oh no, apologies for the instability. :neutral_face: And thanks a lot for reporting it, I'll re-open this issue.\r\n\r\n> Failed to write to 'state.settings'\"\r\n> No ability to save layouts.\r\n> Also the terminals stopped working, no bash.\r\n\r\nThis looks like a connectivity problem to the container. Usually a hard-refresh of the tab solves it (temporary connection problem between front-end and back-end), but maybe sometimes it doesn't, which would mean there is a bug with the IDE itself.\r\n\r\n> 'nak' utility was terminated by signal SIGKILL\r\n\r\nHmm, this doesn't look too dramatic (`nak` is a search module in Cloud9 IDE, but everything should work fine without it) but I'm not sure what's killing it.\r\n\r\nI suspect it maybe related to the Out-Of-Memory errors we've been having lately (when OVH1 uses more than 100% of its 32GB RAM), because in these cases Linux's OOM-killer starts killing processes at random.\r\n\r\nSome suggestions:\r\n\r\n- Please feel free to share specific container IDs (e.g. `355a586a7006c061` or a full URL like https://ovh1.janitor.technology/355a586a7006c061/8089/ide.html) as we may be able to fix any refresh-surviving instability by restarting the container (or worst case rebooting the OVH1 sever, which causes a brief ~5-10min outage, so not too bad)\r\n- If you see Cloud9 IDE specific problems, please feel free to use our alternative IDE Theia (the URL is the same with ends with `/8090/` instead of `/8089/ide.html`, i.e. https://ovh1.janitor.technology/355a586a7006c061/8090/ with our previous example). We'll soon add links to these URLs in our UI, but they should already work fine."}}
{"actor":{"display_login":"jankeromnes"}}