{"action":"closed","issue":{"url":"https://api.github.com/repos/jupyterhub/jupyterhub/issues/1889","repository_url":"https://api.github.com/repos/jupyterhub/jupyterhub","labels_url":"https://api.github.com/repos/jupyterhub/jupyterhub/issues/1889/labels{/name}","comments_url":"https://api.github.com/repos/jupyterhub/jupyterhub/issues/1889/comments","events_url":"https://api.github.com/repos/jupyterhub/jupyterhub/issues/1889/events","html_url":"https://github.com/jupyterhub/jupyterhub/issues/1889","id":322890623,"number":1889,"title":"Hub continually attempts to spawn new proxy after restart with cleanup_proxy=False","user":{"login":"stephenpascoe","id":677427,"avatar_url":"https://avatars0.githubusercontent.com/u/677427?v=4","gravatar_id":"","url":"https://api.github.com/users/stephenpascoe","html_url":"https://github.com/stephenpascoe","followers_url":"https://api.github.com/users/stephenpascoe/followers","following_url":"https://api.github.com/users/stephenpascoe/following{/other_user}","gists_url":"https://api.github.com/users/stephenpascoe/gists{/gist_id}","starred_url":"https://api.github.com/users/stephenpascoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stephenpascoe/subscriptions","organizations_url":"https://api.github.com/users/stephenpascoe/orgs","repos_url":"https://api.github.com/users/stephenpascoe/repos","events_url":"https://api.github.com/users/stephenpascoe/events{/privacy}","received_events_url":"https://api.github.com/users/stephenpascoe/received_events","type":"User","site_admin":false},"labels":[{"id":106589228,"url":"https://api.github.com/repos/jupyterhub/jupyterhub/labels/question","name":"question","color":"Cfaef7","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-05-14T16:29:38Z","updated_at":"2018-05-15T11:03:12Z","closed_at":"2018-05-15T11:03:12Z","author_association":"NONE","body":"\r\n**How to reproduce the issue**\r\n\r\n - Configure jupyterhub with `cleanup_proxy=False`\r\n - Kill jupyterhub process with SIGINT\r\n - Restart jupyterhub\r\n\r\n**What you expected to happen**\r\nService continues as normal after restart\r\n\r\n**What actually happens**\r\nThe proxy process stays running and is now not a subprocess of the hub.  The hub continually attempts to restart the proxy (as shown in the logs).  The frequency of this restart accelerates until it is >1 per second, the hub consumes 100% of CPU and becomes unresponsive.\r\n\r\n\r\n**Share what version of JupyterHub you are using**\r\n```\r\n$ jupyterhub --version\r\n0.8.1\r\n$ configurable-http-proxy --version\r\n3.1.1\r\n```\r\n\r\n\r\nRunning `jupyter troubleshoot` from the command line, if possible, and posting\r\nits output would also be helpful.\r\n\r\nI don't see anything helpful here\r\n\r\n```\r\nebooksdev00:/nfs/vnx-home/spascoe# /usr/local/bin/jupyterhub -f /opt/jupyterhub/jupyterhub_config.py troubleshoot\r\n[I 2018-05-14 17:13:23.199 JupyterHub app:830] Loading cookie_secret from env[JPY_COOKIE_SECRET]\r\n[I 2018-05-14 17:13:23.316 JupyterHub app:1528] Hub API listening on http://127.0.0.1:8081/hub/\r\n[W 2018-05-14 17:13:23.317 JupyterHub proxy:456] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...\r\n[I 2018-05-14 17:13:23.317 JupyterHub proxy:458] Starting proxy @ http://*:8000/\r\n17:13:23.442 - info: [ConfigProxy] Proxying http://*:8000 to (no default)\r\n17:13:23.445 - info: [ConfigProxy] Proxy API at http://127.0.0.1:8001/api/routes\r\n[I 2018-05-14 17:13:23.562 JupyterHub app:1544] Starting managed service cull-idle\r\n[I 2018-05-14 17:13:23.563 JupyterHub service:266] Starting service 'cull-idle': ['python', '/opt/jupyterhub//cull_idle_servers.py', '--timeout=43200']\r\n[I 2018-05-14 17:13:23.567 JupyterHub service:109] Spawning python /opt/jupyterhub//cull_idle_servers.py --timeout=43200\r\n17:13:23.579 - info: [ConfigProxy] 200 GET /api/routes\r\n[W 2018-05-14 17:13:23.580 JupyterHub proxy:304] Adding missing default route\r\n[I 2018-05-14 17:13:23.580 JupyterHub proxy:370] Adding default route for Hub: / => http://127.0.0.1:8081\r\n17:13:23.624 - info: [ConfigProxy] Adding route / -> http://127.0.0.1:8081\r\n17:13:23.625 - info: [ConfigProxy] 201 POST /api/routes/\r\n[I 2018-05-14 17:13:23.626 JupyterHub app:1581] JupyterHub is now running at http://:8000/\r\n[I 2018-05-14 17:13:23.648 JupyterHub log:122] 200 GET /hub/api/users (cull-idle@127.0.0.1) 7.82ms\r\n\r\n\r\n[I 2018-05-14 17:13:49.065 JupyterHub log:122] 200 GET /hub/home (spascoe@::ffff:10.141.30.228) 20.09ms\r\n[I 2018-05-14 17:13:51.849 JupyterHub log:122] 200 GET /hub/spawn (spascoe@::ffff:10.141.30.228) 24.88ms\r\nLast login: Mon May 14 16:35:01 BST 2018 from 10.160.74.34 on pts/0\r\n\r\n\r\n[I 2018-05-14 17:13:56.679 JupyterHub spawner:978] Spawning jupyter-labhub --port=52268 '--notebook-dir=\"/notebook\"' '--NotebookApp.default_url=\"/lab\"' --FileContentsManager.delete_to_trash=False\r\n[I 2018-05-14 17:14:02.450 SingleUserLabApp __init__:250] [jupyter_nbextensions_configurator] enabled 0.2.8\r\n[I 2018-05-14 17:14:02.481 SingleUserLabApp extension:53] JupyterLab beta preview extension loaded from /usr/local/lib/python3.5/dist-packages/jupyterlab\r\n[I 2018-05-14 17:14:02.481 SingleUserLabApp extension:54] JupyterLab application directory is /usr/local/share/jupyter/lab\r\n[I 2018-05-14 17:14:02.546 SingleUserLabApp singleuser:365] Starting jupyterhub-singleuser server version 0.8.1\r\n[I 2018-05-14 17:14:02.551 JupyterHub log:122] 200 GET /hub/api (@127.0.0.1) 0.68ms\r\n[I 2018-05-14 17:14:02.552 SingleUserLabApp notebookapp:1585] Serving notebooks from local directory: /notebook\r\n[I 2018-05-14 17:14:02.552 SingleUserLabApp notebookapp:1585] 0 active kernels\r\n[I 2018-05-14 17:14:02.552 SingleUserLabApp notebookapp:1585] The Jupyter Notebook is running at:\r\n[I 2018-05-14 17:14:02.552 SingleUserLabApp notebookapp:1585] http://127.0.0.1:52268/user/spascoe/?token=8357f18e95394bbf8fc811ff4bd2bd55\r\n[I 2018-05-14 17:14:02.552 SingleUserLabApp notebookapp:1586] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).\r\n[C 2018-05-14 17:14:02.557 SingleUserLabApp notebookapp:1626]\r\n\r\n    Copy/paste this URL into your browser when you connect for the first time,\r\n    to login with a token:\r\n        http://127.0.0.1:52268/user/spascoe/?token=8357f18e95394bbf8fc811ff4bd2bd55\r\n[I 2018-05-14 17:14:03.955 SingleUserLabApp log:122] 302 GET /user/spascoe/ → /user/spascoe/lab? (@127.0.0.1) 0.80ms\r\n[I 2018-05-14 17:14:03.956 JupyterHub base:447] User spascoe took 7.474 seconds to start\r\n[I 2018-05-14 17:14:03.957 JupyterHub proxy:231] Adding user spascoe to proxy /user/spascoe/ => http://127.0.0.1:52268\r\n17:14:03.996 - info: [ConfigProxy] Adding route /user/spascoe -> http://127.0.0.1:52268\r\n17:14:03.996 - info: [ConfigProxy] 201 POST /api/routes/user/spascoe\r\n[I 2018-05-14 17:14:04.001 JupyterHub log:122] 302 POST /hub/spawn → /user/spascoe/ (spascoe@::ffff:10.141.30.228) 7521.26ms\r\n[I 2018-05-14 17:14:04.010 SingleUserLabApp log:122] 302 GET /user/spascoe/ → /user/spascoe/lab? (@::ffff:10.141.30.228) 0.52ms\r\n[I 2018-05-14 17:14:04.018 SingleUserLabApp log:122] 302 GET /user/spascoe/lab? → /hub/api/oauth2/authorize?response_type=code&redirect_uri=%2Fuser%2Fspascoe%2Foauth_callback&client_id=user-spascoe&state=eyJ1dWlkIjogImE0NTdmYzI4MTJmNjQ2MTBhMGEyODUxMjZkYTdkNmFlIiwgIm5leHRfdXJsIjogIi91c2VyL3NwYXNjb2UvbGFiPyJ9 (@::ffff:10.141.30.228) 1.82ms\r\n[I 2018-05-14 17:14:04.036 JupyterHub log:122] 302 GET /hub/api/oauth2/authorize?response_type=code&redirect_uri=%2Fuser%2Fspascoe%2Foauth_callback&client_id=user-spascoe&state=eyJ1dWlkIjogImE0NTdmYzI4MTJmNjQ2MTBhMGEyODUxMjZkYTdkNmFlIiwgIm5leHRfdXJsIjogIi91c2VyL3NwYXNjb2UvbGFiPyJ9 → /user/spascoe/oauth_callback?code=984ebaa6-4263-4935-9a11-e53ea04e80dc&state=eyJ1dWlkIjogImE0NTdmYzI4MTJmNjQ2MTBhMGEyODUxMjZkYTdkNmFlIiwgIm5leHRfdXJsIjogIi91c2VyL3NwYXNjb2UvbGFiPyJ9 (spascoe@::ffff:10.141.30.228) 8.96ms\r\n[I 2018-05-14 17:14:04.075 JupyterHub log:122] 200 POST /hub/api/oauth2/token (spascoe@127.0.0.1) 16.90ms\r\n[I 2018-05-14 17:14:04.081 JupyterHub log:122] 200 GET /hub/api/authorizations/token/[secret] (spascoe@127.0.0.1) 4.05ms\r\n[I 2018-05-14 17:14:04.081 SingleUserLabApp auth:818] Logged-in user {'kind': 'user', 'name': 'spascoe', 'server': '/user/spascoe/', 'pending': None, 'groups': [], 'last_activity': '2018-05-14T16:13:57.760912', 'admin': True}\r\n[I 2018-05-14 17:14:04.082 SingleUserLabApp log:122] 302 GET /user/spascoe/oauth_callback?code=984ebaa6-4263-4935-9a11-e53ea04e80dc&state=eyJ1dWlkIjogImE0NTdmYzI4MTJmNjQ2MTBhMGEyODUxMjZkYTdkNmFlIiwgIm5leHRfdXJsIjogIi91c2VyL3NwYXNjb2UvbGFiPyJ9 → /user/spascoe/lab? (@::ffff:10.141.30.228) 40.53ms\r\n[I 2018-05-14 17:14:04.096 JupyterHub log:122] 200 GET /hub/api/authorizations/token/[secret] (spascoe@127.0.0.1) 2.67ms\r\n[I 2018-05-14 17:14:04.111 SingleUserLabApp log:122] 200 GET /user/spascoe/lab? (spascoe@::ffff:10.141.30.228) 21.11ms\r\n[I 2018-05-14 17:14:05.059 JupyterHub log:122] 200 GET /hub/api/authorizations/token/[secret] (spascoe@127.0.0.1) 3.50ms\r\n[I 2018-05-14 17:14:05.126 SingleUserLabApp log:122] 200 GET /user/spascoe/api/kernelspecs?1526314445047 (spascoe@::ffff:10.141.30.228) 74.03ms\r\n[I 2018-05-14 17:14:05.128 SingleUserLabApp log:122] 200 GET /user/spascoe/api/terminals?1526314445048 (spascoe@::ffff:10.141.30.228) 0.95ms\r\n[I 2018-05-14 17:14:05.150 SingleUserLabApp log:122] 200 GET /user/spascoe/lab/api/settings/@jupyterlab/shortcuts-extension:plugin?1526314445092 (spascoe@::ffff:10.141.30.228) 20.63ms\r\n[I 2018-05-14 17:14:05.168 SingleUserLabApp log:122] 200 GET /user/spascoe/lab/api/settings/@jupyterlab/docmanager-extension:plugin?1526314445114 (spascoe@::ffff:10.141.30.228) 15.47ms\r\n[I 2018-05-14 17:14:05.183 SingleUserLabApp log:122] 200 GET /user/spascoe/lab/api/settings/@jupyterlab/apputils-extension:themes?1526314445120 (spascoe@::ffff:10.141.30.228) 13.62ms\r\n[I 2018-05-14 17:14:05.192 SingleUserLabApp log:122] 200 GET /user/spascoe/api/contents/?content=1&1526314445124 (spascoe@::ffff:10.141.30.228) 8.60ms\r\n[I 2018-05-14 17:14:05.196 SingleUserLabApp log:122] 200 GET /user/spascoe/lab/api/settings/@jupyterlab/notebook-extension:tracker?1526314445133 (spascoe@::ffff:10.141.30.228) 3.44ms\r\n[I 2018-05-14 17:14:05.200 SingleUserLabApp log:122] 200 GET /user/spascoe/lab/api/settings/@jupyterlab/fileeditor-extension:plugin?1526314445136 (spascoe@::ffff:10.141.30.228) 2.91ms\r\n[I 2018-05-14 17:14:05.232 SingleUserLabApp log:122] 200 GET /user/spascoe/lab/api/settings/@jupyterlab/codemirror-extension:commands?1526314445222 (spascoe@::ffff:10.141.30.228) 7.47ms\r\n[I 2018-05-14 17:14:05.246 SingleUserLabApp log:122] 200 GET /user/spascoe/api/terminals?1526314445243 (spascoe@::ffff:10.141.30.228) 1.09ms\r\n[I 2018-05-14 17:14:05.425 SingleUserLabApp log:122] 200 GET /user/spascoe/api/sessions?1526314445419 (spascoe@::ffff:10.141.30.228) 2.03ms\r\n[I 2018-05-14 17:14:05.588 SingleUserLabApp log:122] 200 GET /user/spascoe/api/contents/personal/spascoe?1526314445578 (spascoe@::ffff:10.141.30.228) 4.74ms\r\n[I 2018-05-14 17:14:05.591 SingleUserLabApp log:122] 200 GET /user/spascoe/api/contents/personal/spascoe/Untitled.ipynb?content=0&1526314445582 (spascoe@::ffff:10.141.30.228) 1.84ms\r\n[I 2018-05-14 17:14:05.604 SingleUserLabApp log:122] 200 GET /user/spascoe/api/contents/personal/spascoe?content=1&1526314445599 (spascoe@::ffff:10.141.30.228) 2.20ms\r\n[I 2018-05-14 17:14:06.027 SingleUserLabApp log:122] 200 GET /user/spascoe/api/contents/personal/spascoe/Untitled.ipynb?type=notebook&content=1&1526314445949 (spascoe@::ffff:10.141.30.228) 75.53ms\r\n[I 2018-05-14 17:14:06.163 SingleUserLabApp build_handler:43] Build is up to date\r\n[I 2018-05-14 17:14:06.165 SingleUserLabApp log:122] 200 GET /user/spascoe/lab/api/build?1526314445110 (spascoe@::ffff:10.141.30.228) 1014.04ms\r\n[I 2018-05-14 17:14:06.606 SingleUserLabApp log:122] 200 GET /user/spascoe/api/contents/personal/spascoe/Untitled.ipynb/checkpoints?1526314446598 (spascoe@::ffff:10.141.30.228) 2.15ms\r\n[I 2018-05-14 17:14:06.608 SingleUserLabApp log:122] 200 GET /user/spascoe/api/contents/personal/spascoe/Untitled.ipynb/checkpoints?1526314446598 (spascoe@::ffff:10.141.30.228) 0.71ms\r\n[I 2018-05-14 17:14:06.609 SingleUserLabApp log:122] 200 GET /user/spascoe/api/contents/personal/spascoe/Untitled.ipynb/checkpoints?1526314446600 (spascoe@::ffff:10.141.30.228) 0.77ms\r\n[I 2018-05-14 17:14:06.757 SingleUserLabApp kernelmanager:152] Kernel started: 7bf3a328-ca34-41e5-bf59-6113058ae7d2\r\n[I 2018-05-14 17:14:06.758 SingleUserLabApp log:122] 201 POST /user/spascoe/api/sessions?1526314446702 (spascoe@::ffff:10.141.30.228) 52.62ms\r\n[I 2018-05-14 17:14:07.409 SingleUserLabApp handlers:189] Adapting to protocol v5.1 for kernel 7bf3a328-ca34-41e5-bf59-6113058ae7d2\r\n[I 2018-05-14 17:14:07.411 SingleUserLabApp log:122] 101 GET /user/spascoe/api/kernels/7bf3a328-ca34-41e5-bf59-6113058ae7d2/channels?session_id=141b6521d4cd0ccf6e81c49f4a25d78d&token=8357f18e95394bbf8fc811ff4bd2bd55 (spascoe@::ffff:10.141.30.228) 549.92ms\r\n[I 2018-05-14 17:14:07.431 SingleUserLabApp handlers:189] Adapting to protocol v5.1 for kernel 7bf3a328-ca34-41e5-bf59-6113058ae7d2\r\n[I 2018-05-14 17:14:07.432 SingleUserLabApp log:122] 101 GET /user/spascoe/api/kernels/7bf3a328-ca34-41e5-bf59-6113058ae7d2/channels?session_id=3b3b3173abef5100af40aea165aab659&token=8357f18e95394bbf8fc811ff4bd2bd55 (spascoe@::ffff:10.141.30.228) 2.73ms\r\n[I 2018-05-14 17:14:15.054 SingleUserLabApp log:122] 200 GET /user/spascoe/api/sessions?1526314455049 (spascoe@::ffff:10.141.30.228) 0.95ms\r\n\r\n```\r\n\r\nprobably more relevant is the config file, which is largely unchanged since we upgraded from v0.7.2 of the hub.\r\n```\r\nc.JupyterHub.base_url = '/'\r\n\r\nc.JupyterHub.extra_log_file = '/var/log/jupyterhub.log'\r\nc.Spawner.debug = True\r\nc.LocalProcessSpawner.debug = False\r\nc.JupyterHub.log_level = 'INFO'\r\nc.SudoSpawner.debug_mediator = True\r\n\r\nc.JupyterHub.db_url = \"/opt/jupyterhub//jupyterhub.sqlite\"\r\n\r\nc.JupyterHub.cleanup_proxy = False\r\n\r\nc.JupyterHub.cleanup_servers = False\r\n\r\nc.JupyterHub.logo_file = '/opt/jupyterhub//ont.png'\r\n\r\nc.JupyterHub.services = [\r\n  {\r\n    'name': 'cull-idle',\r\n    'admin': True,\r\n    'command': 'python /opt/jupyterhub//cull_idle_servers.py --timeout=43200'.split(),\r\n  }\r\n]\r\n\r\n\r\nc.JupyterHub.base_url = '/'\r\n\r\nc.JupyterHub.extra_log_file = '/var/log/jupyterhub.log'\r\nc.Spawner.debug = True\r\nc.LocalProcessSpawner.debug = False\r\nc.JupyterHub.log_level = 'INFO'\r\nc.SudoSpawner.debug_mediator = True\r\n\r\nc.JupyterHub.db_url = \"/opt/jupyterhub//jupyterhub.sqlite\"\r\n\r\nc.JupyterHub.cleanup_proxy = False\r\n\r\nc.JupyterHub.cleanup_servers = False\r\n\r\nc.JupyterHub.services = [\r\n  {\r\n    'name': 'cull-idle',\r\n    'admin': True,\r\n    'command': 'python /opt/jupyterhub//cull_idle_servers.py --timeout=43200'.split(),\r\n  }\r\n]\r\nc.Spawner.notebook_dir = '/notebook'\r\n\r\nc.Spawner.args = ['--FileContentsManager.delete_to_trash=False']\r\n\r\n\r\nc.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'\r\nc.ProfilesSpawner.profiles = [\r\n  (\"Traditional\", 'traditional', 'jupyterhub.spawner.LocalProcessSpawner', {}),\r\n  ('JupterLab', 'jupterlab', 'jupyterhub.spawner.LocalProcessSpawner', {'default_url': '/lab', 'cmd': 'jupyter-labhub'}),\r\n]\r\n\r\n```\r\n"}}
{"actor":{"display_login":"stephenpascoe"}}