{"action":"created","issue":{"url":"https://api.github.com/repos/ucbrise/clipper/issues/537","repository_url":"https://api.github.com/repos/ucbrise/clipper","labels_url":"https://api.github.com/repos/ucbrise/clipper/issues/537/labels{/name}","comments_url":"https://api.github.com/repos/ucbrise/clipper/issues/537/comments","events_url":"https://api.github.com/repos/ucbrise/clipper/issues/537/events","html_url":"https://github.com/ucbrise/clipper/issues/537","id":340144034,"node_id":"MDU6SXNzdWUzNDAxNDQwMzQ=","number":537,"title":"Hang several hours:[clipper_admin.py:124] Clipper still initializing.","user":{"login":"xuguangzhao","id":40658303,"node_id":"MDQ6VXNlcjQwNjU4MzAz","avatar_url":"https://avatars3.githubusercontent.com/u/40658303?v=4","gravatar_id":"","url":"https://api.github.com/users/xuguangzhao","html_url":"https://github.com/xuguangzhao","followers_url":"https://api.github.com/users/xuguangzhao/followers","following_url":"https://api.github.com/users/xuguangzhao/following{/other_user}","gists_url":"https://api.github.com/users/xuguangzhao/gists{/gist_id}","starred_url":"https://api.github.com/users/xuguangzhao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xuguangzhao/subscriptions","organizations_url":"https://api.github.com/users/xuguangzhao/orgs","repos_url":"https://api.github.com/users/xuguangzhao/repos","events_url":"https://api.github.com/users/xuguangzhao/events{/privacy}","received_events_url":"https://api.github.com/users/xuguangzhao/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-07-11T08:37:11Z","updated_at":"2018-08-14T11:49:27Z","closed_at":null,"author_association":"NONE","body":"before clipper is correctly running. because access to private registry is need configed for each node, so when i have done for registry ,I restart docker at each node.  so i  deploy the clipper in k8s cluster again. but when i  exec\r\nclipper_admin.start_clipper()， log is always printing Clipper still initializing（the state has several hours）：\r\n\r\nIn [1]: from clipper_admin import ClipperConnection, KubernetesContainerManager\r\n\r\nIn [2]: clipper_conn = ClipperConnection(KubernetesContainerManager(useInternalIP=True))\r\n\r\nIn [3]: clipper_conn.start_clipper()\r\n18-07-11:08:00:42 WARNING  [kubernetes_container_manager.py:231] No external node addresses found. Using Internal IP address\r\n18-07-11:08:00:42 INFO     [kubernetes_container_manager.py:244] Found 1 nodes: 192.168.13.79\r\n18-07-11:08:00:42 INFO     [kubernetes_container_manager.py:253] Setting Clipper mgmt port to 31238\r\n18-07-11:08:00:42 INFO     [kubernetes_container_manager.py:261] Setting Clipper query port to 31048\r\n18-07-11:08:00:42 INFO     [kubernetes_container_manager.py:276] Setting Clipper metric port to 30965\r\n18-07-11:08:00:42 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n18-07-11:08:00:43 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n18-07-11:08:00:44 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n18-07-11:08:00:45 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n18-07-11:08:00:46 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n18-07-11:08:00:47 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n18-07-11:08:00:48 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n18-07-11:08:00:49 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n18-07-11:08:00:50 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n18-07-11:08:00:51 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n18-07-11:08:00:52 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n18-07-11:08:00:53 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n18-07-11:08:00:54 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n18-07-11:08:00:55 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n18-07-11:08:00:56 INFO     [clipper_admin.py:124] Clipper still initializing.\r\n..........\r\n\r\ncurrent environment:\r\n[root@zhiyuan1 ~]# kubectl get pods -o wide\r\nNAME                              READY     STATUS    RESTARTS   AGE       IP             NODE\r\nload-generator-7bbbb4fdd4-zsswb   1/1       Running   5          6d        10.244.3.137   zhiyuan2.novalocal\r\nmetrics-7d577dbc99-lmwwb          1/1       Running   0              6h       10.244.2.161   zhiyuan-3\r\nmgmt-frontend-7858f5c57c-vmhvz    1/1       Running   0         6h       10.244.2.160   zhiyuan-3\r\nphp-apache-c55756996-rjkjs        1/1       Running   4          1d        10.244.2.146   zhiyuan-3\r\nquery-frontend-0-b4df59c5-tgmbr   2/2       Running   0          6h       10.244.4.60    zhiyuan-4\r\nredis-56d457969d-gkl56            1/1       Running   0          6h      10.244.4.59    zhiyuan-4\r\nwebapp-7qkxx                      1/1       Running   3          8d        10.244.3.136   zhiyuan2.novalocal\r\nwebapp-nm9c2                      1/1       Running   3          8d        10.244.0.41    zhiyuan1.novalocal\r\n[root@zhiyuan1 ~]# \r\n\r\n[root@zhiyuan1 ~]# kubectl get service  -o wide\r\nNAME               TYPE        CLUSTER-IP       EXTERNAL-IP     PORT(S)          AGE       SELECTOR\r\nkubernetes         ClusterIP   10.96.0.1        <none>          443/TCP          14d       <none>\r\nmetrics            NodePort    10.96.26.77      <none>          9090:30965/TCP   6h      ai.clipper.name=metrics\r\nmgmt-frontend      NodePort    10.96.39.108     <none>          1338:31238/TCP   6h     ai.clipper.name=mgmt-frontend\r\nphp-apache         ClusterIP   10.106.230.133   192.168.13.82   80/TCP           6d        run=php-apache\r\nquery-frontend     NodePort    10.108.199.22    <none>          1337:31048/TCP   6h       ai.clipper.name=query-frontend\r\nquery-frontend-0   NodePort    10.100.205.246   <none>          7000:32480/TCP   6h     ai.clipper.name=query-frontend,ai.clipper.query_frontend.id=0\r\nredis              NodePort    10.97.136.40     <none>          6379:31481/TCP   6h       ai.clipper.name=redis\r\nwebapp             NodePort    10.102.224.207   <none>          8081:30003/TCP   9d        app=webapp\r\n[root@zhiyuan1 ~]# \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/ucbrise/clipper/issues/comments/412845562","html_url":"https://github.com/ucbrise/clipper/issues/537#issuecomment-412845562","issue_url":"https://api.github.com/repos/ucbrise/clipper/issues/537","id":412845562,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMjg0NTU2Mg==","user":{"login":"ksoucy","id":39825513,"node_id":"MDQ6VXNlcjM5ODI1NTEz","avatar_url":"https://avatars1.githubusercontent.com/u/39825513?v=4","gravatar_id":"","url":"https://api.github.com/users/ksoucy","html_url":"https://github.com/ksoucy","followers_url":"https://api.github.com/users/ksoucy/followers","following_url":"https://api.github.com/users/ksoucy/following{/other_user}","gists_url":"https://api.github.com/users/ksoucy/gists{/gist_id}","starred_url":"https://api.github.com/users/ksoucy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ksoucy/subscriptions","organizations_url":"https://api.github.com/users/ksoucy/orgs","repos_url":"https://api.github.com/users/ksoucy/repos","events_url":"https://api.github.com/users/ksoucy/events{/privacy}","received_events_url":"https://api.github.com/users/ksoucy/received_events","type":"User","site_admin":false},"created_at":"2018-08-14T11:49:27Z","updated_at":"2018-08-14T11:49:27Z","author_association":"NONE","body":"I also experience the same issue but using k8s locally with minikube."}}
{"actor":{"display_login":"ksoucy"}}