{"action":"created","issue":{"url":"https://api.github.com/repos/camsas/poseidon/issues/1","repository_url":"https://api.github.com/repos/camsas/poseidon","labels_url":"https://api.github.com/repos/camsas/poseidon/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/camsas/poseidon/issues/1/comments","events_url":"https://api.github.com/repos/camsas/poseidon/issues/1/events","html_url":"https://github.com/camsas/poseidon/issues/1","id":193398116,"number":1,"title":"containerized poseidon: memory corruption","user":{"login":"dalanlan","id":5979704,"avatar_url":"https://avatars.githubusercontent.com/u/5979704?v=3","gravatar_id":"","url":"https://api.github.com/users/dalanlan","html_url":"https://github.com/dalanlan","followers_url":"https://api.github.com/users/dalanlan/followers","following_url":"https://api.github.com/users/dalanlan/following{/other_user}","gists_url":"https://api.github.com/users/dalanlan/gists{/gist_id}","starred_url":"https://api.github.com/users/dalanlan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dalanlan/subscriptions","organizations_url":"https://api.github.com/users/dalanlan/orgs","repos_url":"https://api.github.com/users/dalanlan/repos","events_url":"https://api.github.com/users/dalanlan/events{/privacy}","received_events_url":"https://api.github.com/users/dalanlan/received_events","type":"User","site_admin":false},"labels":[{"id":270826216,"url":"https://api.github.com/repos/camsas/poseidon/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-12-05T02:31:58Z","updated_at":"2016-12-05T03:43:41Z","closed_at":null,"body":"@ms705 Hi there!\r\nThank you so much for your amazing work (again). I've tried to bring up poseidon in docker, yet it seems to suffer some memory issue after it started to run for 2~3 min. \r\nPlease kindly take a look at the log if possible. Thanks so much.\r\n\r\n```\r\nroot@ubuntu:~# docker logs 0f0e6ab6837e\r\nI1205 02:11:00.477520     1 k8s_api_client.cc:47] Starting K8sApiClient for API server at http://10.10.103.67:8080/\r\nI1205 02:11:00.481966     1 scheduler_integration.cc:134] Firmament scheduler instantiated: <FlowScheduler for coordinator >\r\nI1205 02:11:00.526311     1 scheduler_integration.cc:145] Adding new node's resource with RID cf742c5b-2ef6-4b18-a93d-3785f5444d22\r\nI1205 02:11:00.547468     1 scheduler_integration.cc:171] New unscheduled pod: busybox\r\nI1205 02:11:00.548526     1 utils.cc:324] External execution of command: /usr/bin/cs2.exe \r\nI1205 02:11:00.662428     1 scheduler_integration.cc:189] Got 1 scheduling deltas\r\nI1205 02:11:00.663835     1 scheduler_integration.cc:191] Delta: task_id: 6569851536870938357\r\nresource_id: \"cf742c5b-2ef6-4b18-a93d-3785f5444d22\"\r\ntype: PLACE\r\nI1205 02:11:00.672199    40 k8s_api_client.cc:72] Parsing binding response: {\"apiVersion\":\"v1\",\"code\":409,\"details\":{\"kind\":\"pods/binding\",\"name\":\"busybox\"},\"kind\":\"Status\",\"message\":\"Operation cannot be fulfilled on pods/binding \\\"busybox\\\": pod busybox is already assigned to node \\\"10.10.103.69\\\"\",\"metadata\":{},\"reason\":\"Conflict\",\"status\":\"Failure\"}\r\nI1205 02:11:00.673027     1 k8s_api_client.cc:201] Bound busybox to 10.10.103.67\r\nI1205 02:11:10.695752     1 scheduler_integration.cc:171] New unscheduled pod: k8s-master-10.10.103.67\r\nI1205 02:11:10.696748     1 utils.cc:324] External execution of command: /usr/bin/cs2.exe \r\nI1205 02:11:10.802476     1 scheduler_integration.cc:189] Got 1 scheduling deltas\r\nI1205 02:11:10.803570     1 scheduler_integration.cc:191] Delta: task_id: 8144958691656263322\r\nresource_id: \"cf742c5b-2ef6-4b18-a93d-3785f5444d22\"\r\ntype: PLACE\r\nI1205 02:11:10.812801    46 k8s_api_client.cc:72] Parsing binding response: {\"apiVersion\":\"v1\",\"code\":404,\"details\":{\"kind\":\"pods\",\"name\":\"k8s-master-10.10.103.67\"},\"kind\":\"Status\",\"message\":\"pods \\\"k8s-master-10.10.103.67\\\" not found\",\"metadata\":{},\"reason\":\"NotFound\",\"status\":\"Failure\"}\r\nI1205 02:11:10.814126     1 k8s_api_client.cc:201] Bound k8s-master-10.10.103.67 to 10.10.103.67\r\nI1205 02:11:20.839632     1 scheduler_integration.cc:171] New unscheduled pod: k8s-proxy-v1-7kvf0\r\nI1205 02:11:20.840548     1 utils.cc:324] External execution of command: /usr/bin/cs2.exe \r\nI1205 02:11:20.952191     1 scheduler_integration.cc:189] Got 1 scheduling deltas\r\nI1205 02:11:20.952741     1 scheduler_integration.cc:191] Delta: task_id: 7995058044728560469\r\nresource_id: \"cf742c5b-2ef6-4b18-a93d-3785f5444d22\"\r\ntype: PLACE\r\nI1205 02:11:20.964922    31 k8s_api_client.cc:72] Parsing binding response: {\"apiVersion\":\"v1\",\"code\":404,\"details\":{\"kind\":\"pods\",\"name\":\"k8s-proxy-v1-7kvf0\"},\"kind\":\"Status\",\"message\":\"pods \\\"k8s-proxy-v1-7kvf0\\\" not found\",\"metadata\":{},\"reason\":\"NotFound\",\"status\":\"Failure\"}\r\nI1205 02:11:20.965868     1 k8s_api_client.cc:201] Bound k8s-proxy-v1-7kvf0 to 10.10.103.67\r\nI1205 02:11:30.994323     1 scheduler_integration.cc:171] New unscheduled pod: k8s-proxy-v1-eahvt\r\nI1205 02:11:30.995163     1 utils.cc:324] External execution of command: /usr/bin/cs2.exe \r\nI1205 02:11:31.108824     1 scheduler_integration.cc:189] Got 1 scheduling deltas\r\nI1205 02:11:31.109328     1 scheduler_integration.cc:191] Delta: task_id: 13663010032745704128\r\nresource_id: \"cf742c5b-2ef6-4b18-a93d-3785f5444d22\"\r\ntype: PLACE\r\nI1205 02:11:31.118139    34 k8s_api_client.cc:72] Parsing binding response: {\"apiVersion\":\"v1\",\"code\":404,\"details\":{\"kind\":\"pods\",\"name\":\"k8s-proxy-v1-eahvt\"},\"kind\":\"Status\",\"message\":\"pods \\\"k8s-proxy-v1-eahvt\\\" not found\",\"metadata\":{},\"reason\":\"NotFound\",\"status\":\"Failure\"}\r\nI1205 02:11:31.118768     1 k8s_api_client.cc:201] Bound k8s-proxy-v1-eahvt to 10.10.103.67\r\nI1205 02:11:41.136282     1 scheduler_integration.cc:171] New unscheduled pod: k8s-proxy-v1-qce6o\r\nI1205 02:11:41.136919     1 utils.cc:324] External execution of command: /usr/bin/cs2.exe \r\nI1205 02:11:41.192404     1 scheduler_integration.cc:189] Got 1 scheduling deltas\r\nI1205 02:11:41.193066     1 scheduler_integration.cc:191] Delta: task_id: 18392520999753559797\r\nresource_id: \"cf742c5b-2ef6-4b18-a93d-3785f5444d22\"\r\ntype: PLACE\r\nI1205 02:11:41.199827    34 k8s_api_client.cc:72] Parsing binding response: {\"apiVersion\":\"v1\",\"code\":404,\"details\":{\"kind\":\"pods\",\"name\":\"k8s-proxy-v1-qce6o\"},\"kind\":\"Status\",\"message\":\"pods \\\"k8s-proxy-v1-qce6o\\\" not found\",\"metadata\":{},\"reason\":\"NotFound\",\"status\":\"Failure\"}\r\nI1205 02:11:41.199950     1 k8s_api_client.cc:201] Bound k8s-proxy-v1-qce6o to 10.10.103.67\r\nI1205 02:11:51.220187     1 scheduler_integration.cc:171] New unscheduled pod: kube-addon-manager-10.10.103.67\r\nI1205 02:11:51.220731     1 utils.cc:324] External execution of command: /usr/bin/cs2.exe \r\nI1205 02:11:51.331187     1 scheduler_integration.cc:189] Got 1 scheduling deltas\r\nI1205 02:11:51.331946     1 scheduler_integration.cc:191] Delta: task_id: 7867583654750718473\r\nresource_id: \"cf742c5b-2ef6-4b18-a93d-3785f5444d22\"\r\ntype: PLACE\r\nI1205 02:11:51.339476    41 k8s_api_client.cc:72] Parsing binding response: {\"apiVersion\":\"v1\",\"code\":404,\"details\":{\"kind\":\"pods\",\"name\":\"kube-addon-manager-10.10.103.67\"},\"kind\":\"Status\",\"message\":\"pods \\\"kube-addon-manager-10.10.103.67\\\" not found\",\"metadata\":{},\"reason\":\"NotFound\",\"status\":\"Failure\"}\r\nI1205 02:11:51.339663     1 k8s_api_client.cc:201] Bound kube-addon-manager-10.10.103.67 to 10.10.103.67\r\nI1205 02:12:01.368042     1 scheduler_integration.cc:171] New unscheduled pod: kube-dns-v20-o4p9l\r\nI1205 02:12:01.369421     1 utils.cc:324] External execution of command: /usr/bin/cs2.exe \r\nI1205 02:12:01.494426     1 scheduler_integration.cc:189] Got 1 scheduling deltas\r\nI1205 02:12:01.496582     1 scheduler_integration.cc:191] Delta: task_id: 829239843783619468\r\nresource_id: \"cf742c5b-2ef6-4b18-a93d-3785f5444d22\"\r\ntype: PLACE\r\nI1205 02:12:01.509184    45 k8s_api_client.cc:72] Parsing binding response: {\"apiVersion\":\"v1\",\"code\":404,\"details\":{\"kind\":\"pods\",\"name\":\"kube-dns-v20-o4p9l\"},\"kind\":\"Status\",\"message\":\"pods \\\"kube-dns-v20-o4p9l\\\" not found\",\"metadata\":{},\"reason\":\"NotFound\",\"status\":\"Failure\"}\r\nI1205 02:12:01.509935     1 k8s_api_client.cc:201] Bound kube-dns-v20-o4p9l to 10.10.103.67\r\nI1205 02:12:11.530864     1 scheduler_integration.cc:171] New unscheduled pod: kubernetes-dashboard-v1.4.0-cblag\r\nI1205 02:12:11.531860     1 utils.cc:324] External execution of command: /usr/bin/cs2.exe \r\n*** Error in `/usr/bin/poseidon': malloc(): memory corruption: 0x00007f5a578bd010 ***\r\n```"},"comment":{"url":"https://api.github.com/repos/camsas/poseidon/issues/comments/264761720","html_url":"https://github.com/camsas/poseidon/issues/1#issuecomment-264761720","issue_url":"https://api.github.com/repos/camsas/poseidon/issues/1","id":264761720,"user":{"login":"ms705","id":192315,"avatar_url":"https://avatars.githubusercontent.com/u/192315?v=3","gravatar_id":"","url":"https://api.github.com/users/ms705","html_url":"https://github.com/ms705","followers_url":"https://api.github.com/users/ms705/followers","following_url":"https://api.github.com/users/ms705/following{/other_user}","gists_url":"https://api.github.com/users/ms705/gists{/gist_id}","starred_url":"https://api.github.com/users/ms705/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ms705/subscriptions","organizations_url":"https://api.github.com/users/ms705/orgs","repos_url":"https://api.github.com/users/ms705/repos","events_url":"https://api.github.com/users/ms705/events{/privacy}","received_events_url":"https://api.github.com/users/ms705/received_events","type":"User","site_admin":false},"created_at":"2016-12-05T03:43:41Z","updated_at":"2016-12-05T03:43:41Z","body":"@dalanlan Thanks for reporting this! The memory corruption definitely looks scary, let's try to get to the bottom of this.\r\n\r\nA couple of questions on your setup:\r\n 1. Did you build the Poseidon image yourself, or did you download the image from DockerHub?\r\n 2. What cluster setup and workload are you running? I can see some k8s internal pods in the log, but also busybox. Are all these pods pre-existing, or did you submit new ones during this run?\r\n 3. Is the bug deterministic, i.e., does is happen the same way every time?\r\n\r\nMoreover, there seem to be several problems here:\r\n 1. The memory corruption, obviously.\r\n 2. The fact that Poseidon fails to bind the pods because it receives a 404 error from the k8s API server when trying to do so.\r\n\r\nThe second issue could be related to `kube-scheduler` also running in the environment, but the memory corruption ought not be related.\r\n\r\nThe best way to debug the memory corruption is to make a debug build of Poseidon. To do this, follow these steps in a blank checkout:\r\n```\r\n$ mkdir build\r\n$ cd build\r\n$ cmake .. -DDEBUG=1\r\n$ make\r\n```\r\nThis will take a while, but should build Poseidon, Firmament and all of their dependencies with debug symbols and address sanitizers.\r\n\r\nOnce done, run your experiment again and see if it produces more useful output when the memory corruption happens.\r\n\r\nIf that doesn't help, you can run Poseidon in `gdb`:\r\n```\r\n$ docker run camsas/poseidon:dev /bin/bash\r\ncontainer# apt-get update\r\ncontainer# apt-get install build-essential gdb\r\ncontainer# gdb --args /usr/bin/poseidon \\\r\n    --logtostderr \\\r\n    --k8s_apiserver_host=<host> \\\r\n    --k8s_apiserver_port=<port> \\\r\n    --cs2_binary=/usr/bin/cs2.exe \\\r\n    --max_tasks_per_pu=<max pods per node>\r\n```\r\nThis will run Poseidon in `gdb` and should give you more information at the point of crashing."}}
{"actor":{"display_login":"ms705"}}