{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/25880","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/25880/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/25880/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/25880/events","html_url":"https://github.com/kubernetes/kubernetes/issues/25880","id":155749254,"number":25880,"title":"UnsafeCconvertToVersion seem to cause apiserver crashes","user":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/CSI-API%20Machinery%20SIG","name":"team/CSI-API Machinery SIG","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-05-19T14:22:56Z","updated_at":"2016-05-19T14:23:34Z","closed_at":null,"body":"I've just seen the following crash of apiserver\r\n\r\n    fatal error: I0518 16:49:34.663726    3519 handlers.go:161] PUT /api/v1/namespaces/e2e-tests-density-c37gb/pods/density-latency-pod-1514-xph9f/status: (175.155856ms) 200 [[kubemar        k/v1.3.0 (linux/amd64) kubernetes/952e830] 104.154.120.247:40256]\r\n    concurrent map writes\r\n\r\nFrom the go-routines logs it seems that the victim ones are:\r\n\r\n    5953382 goroutine 25813550 [running]:\r\n    5953383 runtime.throw(0x2dab540, 0x15)\r\n    5953384   /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc8f10cff90 sp=0xc8f10cff78\r\n    5953385 runtime.mapdelete(0x1f2ef60, 0xc86825cb40, 0xc8f10d0078)\r\n    5953386   /usr/local/go/src/runtime/hashmap.go:559 +0x5a fp=0xc8f10cfff0 sp=0xc8f10cff90\r\n    5953387 k8s.io/kubernetes/pkg/api/v1.Convert_api_Pod_To_v1_Pod(0xc914c6c4b0, 0xc914d6d268, 0x7f6837340900, 0xc8f4398e10, 0x0, 0x0)\r\n    5953388   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/v1/conversion.go:536 +0x5c9 fp=0xc8f10d00d0 sp=0xc8f10cfff0\r\n    5953389 k8s.io/kubernetes/pkg/api/v1.autoConvert_api_PodList_To_v1_PodList(0xc8dac18ba0, 0xc8e7bf20c0, 0x7f6837340900, 0xc8f4398e10, 0x0, 0x0)\r\n    5953390   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/v1/conversion_generated.go:4834 +0x31a fp=0xc8f10d01d8 sp=0xc8f10d00d0\r\n    5953391 k8s.io/kubernetes/pkg/api/v1.Convert_api_PodList_To_v1_PodList(0xc8dac18ba0, 0xc8e7bf20c0, 0x7f6837340900, 0xc8f4398e10, 0x0, 0x0)\r\n    5953392   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/v1/conversion_generated.go:4845 +0x4b fp=0xc8f10d0210 sp=0xc8f10d01d8\r\n    5953393 runtime.call64(0xc8206778c0, 0x32c6778, 0xc840badaa0, 0x2000000030)\r\n    5953394   /usr/local/go/src/runtime/asm_amd64.s:473 +0x3e fp=0xc8f10d0258 sp=0xc8f10d0210\r\n    5953395 reflect.Value.call(0x23ce0c0, 0x32c6778, 0x13, 0x2b79030, 0x4, 0xc8f10d0778, 0x3, 0x3, 0x0, 0x0, ...)\r\n    5953396   /usr/local/go/src/reflect/value.go:435 +0x120d fp=0xc8f10d05a8 sp=0xc8f10d0258\r\n    5953397 reflect.Value.Call(0x23ce0c0, 0x32c6778, 0x13, 0xc8f10d0778, 0x3, 0x3, 0x0, 0x0, 0x0)\r\n    5953398   /usr/local/go/src/reflect/value.go:303 +0xb1 fp=0xc8f10d0608 sp=0xc8f10d05a8\r\n    5953399 k8s.io/kubernetes/pkg/conversion.(*Converter).callCustom(0xc8201cc200, 0x29ac1c0, 0xc8dac18ba0, 0x16, 0x2ab5400, 0xc8e7bf20c0, 0x16, 0x23ce0c0, 0x32c6778, 0x13, ...)\r\n    5953400   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/conversion/converter.go:584 +0x4f4 fp=0xc8f10d07c8 sp=0xc8f10d0608\r\n    5953401 k8s.io/kubernetes/pkg/conversion.(*Converter).convert(0xc8201cc200, 0x27236e0, 0xc8dac18ba0, 0x199, 0x284d180, 0xc8e7bf20c0, 0x199, 0xc8f4398e10, 0x0, 0x0)\r\n    5953402   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/conversion/converter.go:627 +0xb76 fp=0xc8f10d0980 sp=0xc8f10d07c8\r\n    5953403 k8s.io/kubernetes/pkg/conversion.(*Converter).(k8s.io/kubernetes/pkg/conversion.convert)-fm(0x27236e0, 0xc8dac18ba0, 0x199, 0x284d180, 0xc8e7bf20c0, 0x199, 0xc8f4398e10, 0x0, 0x0)\r\n    5953404   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/conversion/converter.go:524 +0x84 fp=0xc8f10d09d8 sp=0xc8f10d0980\r\n    5953405 k8s.io/kubernetes/pkg/conversion.(*Converter).doConversion(0xc8201cc200, 0x29ac1c0, 0xc8dac18ba0, 0x2ab5400, 0xc8e7bf20c0, 0x0, 0xc8eedf8698, 0xc8f10d0c18, 0x0, 0x0)\r\n    5953406   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/conversion/converter.go:561 +0x45e fp=0xc8f10d0b78 sp=0xc8f10d09d8\r\n    5953407 k8s.io/kubernetes/pkg/conversion.(*Converter).Convert(0xc8201cc200, 0x29ac1c0, 0xc8dac18ba0, 0x2ab5400, 0xc8e7bf20c0, 0x0, 0xc8eedf8698, 0x0, 0x0)\r\n    5953408   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/conversion/converter.go:524 +0x28d fp=0xc8f10d0c48 sp=0xc8f10d0b78\r\n    5953409 k8s.io/kubernetes/pkg/runtime.(*Scheme).UnsafeConvertToVersion(0xc8201c4340, 0x7f683733b850, 0xc8dac18ba0, 0x0, 0x0, 0x2b85300, 0x2, 0x0, 0x0, 0x0, ...)\r\n    5953410   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/runtime/scheme.go:586 +0x1540 fp=0xc8f10d10f8 sp=0xc8f10d0c48\r\n    5953411 k8s.io/kubernetes/pkg/runtime.unsafeObjectConvertor.ConvertToVersion(0xc8201c4340, 0x7f683733b850, 0xc8dac18ba0, 0x0, 0x0, 0x2b85300, 0x2, 0x0, 0x0, 0x0, ...)\r\n    5953412   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/runtime/helper.go:61 +0x8e fp=0xc8f10d1158 sp=0xc8f10d10f8\r\n    5953413 k8s.io/kubernetes/pkg/runtime/serializer/versioning.(*codec).EncodeToStream(0xc901634cb0, 0x7f683733b850, 0xc8dac18ba0, 0x7f6837362488, 0xc83df9bc20, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n    5953414   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/runtime/serializer/versioning/versioning.go:268 +0xb4d fp=0xc8f10d1438 sp=0xc8f10d1158\r\n    5953415 k8s.io/kubernetes/pkg/apiserver.writeNegotiated(0x7f683733ffa8, 0xc8206ce150, 0x0, 0x0, 0x2b85300, 0x2, 0x7f6837362120, 0xc83df9bc20, 0xc8e31e90a0, 0xc8, ...)\r\n    5953416   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:443 +0x30e fp=0xc8f10d1520 sp=0xc8f10d1438\r\n    5953417 k8s.io/kubernetes/pkg/apiserver.write(0xc8, 0x0, 0x0, 0x2b85300, 0x2, 0x7f683733ffa8, 0xc8206ce150, 0x7f683733b850, 0xc8dac18ba0, 0x7f6837362120, ...)\r\n    5953418   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:427 +0xb70 fp=0xc8f10d1748 sp=0xc8f10d1520\r\n    5953419 k8s.io/kubernetes/pkg/apiserver.ListResource.func1(0xc8d9ab9110, 0xc8dac18300)\r\n    5953420   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:327 +0xeb3 fp=0xc8f10d1b48 sp=0xc8f10d1748\r\n    5953421 k8s.io/kubernetes/pkg/apiserver/metrics.InstrumentRouteFunc.func1(0xc8d9ab9110, 0xc8dac18300)\r\n    5953422   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:101 +0x25c fp=0xc8f10d1c28 sp=0xc8f10d1b48\r\n    5953423 k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc8201dea20, 0x7f6837361f98, 0xc83df9bc00, 0xc8e31e90a0)\r\n    5953424   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:249 +0xf30 fp=0xc8f10d1de0 sp=0xc8f10d1c28\r\n    5953425 k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).(k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch)-fm(0x7f6837361f98, 0xc83df9bc00, 0xc8e31e90        a0)\r\n    5953426   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:95 +0x3e fp=0xc8f10d1e08 sp=0xc8f10d1de0\r\n    5953427 net/http.HandlerFunc.ServeHTTP(0xc820d49880, 0x7f6837361f98, 0xc83df9bc00, 0xc8e31e90a0)\r\n    5953428   /usr/local/go/src/net/http/server.go:1618 +0x3a fp=0xc8f10d1e28 sp=0xc8f10d1e08\r\n    5953429 net/http.(*ServeMux).ServeHTTP(0xc820416c90, 0x7f6837361f98, 0xc83df9bc00, 0xc8e31e90a0)\r\n    5953430   /usr/local/go/src/net/http/server.go:1910 +0x17d fp=0xc8f10d1e80 sp=0xc8f10d1e28\r\n    5953431 k8s.io/kubernetes/pkg/api.NewRequestContextFilter.func1(0x7f6837361f98, 0xc83df9bc00, 0xc8e31e90a0)\r\n    5953432   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:101 +0x157 fp=0xc8f10d1f08 sp=0xc8f10d1e80\r\n    5953433 net/http.HandlerFunc.ServeHTTP(0xc8203bb8c0, 0x7f6837361f98, 0xc83df9bc00, 0xc8e31e90a0)\r\n    5953434   /usr/local/go/src/net/http/server.go:1618 +0x3a fp=0xc8f10d1f28 sp=0xc8f10d1f08\r\n    5953435 k8s.io/kubernetes/pkg/apiserver.(*timeoutHandler).ServeHTTP.func1(0xc820634a80, 0x7f6837361f58, 0xc83df9bc00, 0xc8e31e90a0, 0xc8dac18120)\r\n    5953436   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:191 +0x91 fp=0xc8f10d1f68 sp=0xc8f10d1f28\r\n    5953437 runtime.goexit()\r\n    5953438   /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8f10d1f70 sp=0xc8f10d1f68\r\n    5953439 created by k8s.io/kubernetes/pkg/apiserver.(*timeoutHandler).ServeHTTP\r\n    5953440   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:193 +0x15b\r\n\r\nand\r\n\r\n    6505750 goroutine 25813385 [runnable]:\r\n    6505751 reflect.mapiterinit(0x2634200, 0xc9235c56b0, 0x195)\r\n    6505752   /usr/local/go/src/runtime/hashmap.go:1014 +0x4a\r\n    6505753 reflect.Value.MapKeys(0x2634200, 0xc8e0514f98, 0x195, 0x0, 0x0, 0x0)\r\n    6505754   /usr/local/go/src/reflect/value.go:1075 +0xec\r\n    6505755 encoding/json.(*mapEncoder).encode(0xc820880378, 0xc8db7ff970, 0x2634200, 0xc8e0514f98, 0x195, 0x1c000)\r\n    6505756   /usr/local/go/src/encoding/json/encode.go:614 +0xd6\r\n    6505757 encoding/json.(*mapEncoder).(encoding/json.encode)-fm(0xc8db7ff970, 0x2634200, 0xc8e0514f98, 0x195, 0x0)\r\n    6505758   /usr/local/go/src/encoding/json/encode.go:632 +0x51\r\n    6505759 encoding/json.(*structEncoder).encode(0xc821031440, 0xc8db7ff970, 0x27ffae0, 0xc8e0514f90, 0x199, 0x1c000)\r\n    6505760   /usr/local/go/src/encoding/json/encode.go:587 +0x2c4\r\n    6505761 encoding/json.(*structEncoder).(encoding/json.encode)-fm(0xc8db7ff970, 0x27ffae0, 0xc8e0514f90, 0x199, 0x0)\r\n    6505762   /usr/local/go/src/encoding/json/encode.go:601 +0x51\r\n    6505763 encoding/json.(*structEncoder).encode(0xc821030930, 0xc8db7ff970, 0x2add560, 0xc8e0514f00, 0x199, 0x1c000)\r\n    6505764   /usr/local/go/src/encoding/json/encode.go:587 +0x2c4\r\n    6505765 encoding/json.(*structEncoder).(encoding/json.encode)-fm(0xc8db7ff970, 0x2add560, 0xc8e0514f00, 0x199, 0x2add500)\r\n    6505766   /usr/local/go/src/encoding/json/encode.go:601 +0x51\r\n    6505767 encoding/json.(*arrayEncoder).encode(0xc820686b90, 0xc8db7ff970, 0x1eddca0, 0xc922064ee0, 0x197, 0x1c000)\r\n    6505768   /usr/local/go/src/encoding/json/encode.go:690 +0x11b\r\n    6505769 encoding/json.(*arrayEncoder).(encoding/json.encode)-fm(0xc8db7ff970, 0x1eddca0, 0xc922064ee0, 0x197, 0x0)\r\n    6505770   /usr/local/go/src/encoding/json/encode.go:697 +0x51\r\n    6505771 encoding/json.(*sliceEncoder).encode(0xc820686b98, 0xc8db7ff970, 0x1eddca0, 0xc922064ee0, 0x197, 0x1c000)\r\n    6505772   /usr/local/go/src/encoding/json/encode.go:667 +0xb0\r\n    6505773 encoding/json.(*sliceEncoder).(encoding/json.encode)-fm(0xc8db7ff970, 0x1eddca0, 0xc922064ee0, 0x197, 0x0)\r\n    6505774   /usr/local/go/src/encoding/json/encode.go:676 +0x51\r\n    6505775 encoding/json.(*structEncoder).encode(0xc82112fbf0, 0xc8db7ff970, 0x2add8e0, 0xc922064eb0, 0x199, 0x1c000)\r\n    6505776   /usr/local/go/src/encoding/json/encode.go:587 +0x2c4\r\n    6505777 encoding/json.(*structEncoder).(encoding/json.encode)-fm(0xc8db7ff970, 0x2add8e0, 0xc922064eb0, 0x199, 0x0)\r\n    6505778   /usr/local/go/src/encoding/json/encode.go:601 +0x51\r\n    6505779 encoding/json.(*structEncoder).encode(0xc82112fb00, 0xc8db7ff970, 0x28c72c0, 0xc922064dc0, 0x199, 0x1bd00)\r\n    6505780   /usr/local/go/src/encoding/json/encode.go:587 +0x2c4\r\n    6505781 encoding/json.(*structEncoder).(encoding/json.encode)-fm(0xc8db7ff970, 0x28c72c0, 0xc922064dc0, 0x199, 0x28c7200)\r\n    6505782   /usr/local/go/src/encoding/json/encode.go:601 +0x51\r\n    6505783 encoding/json.(*arrayEncoder).encode(0xc820686e08, 0xc8db7ff970, 0x1ede780, 0xc914313ba0, 0x197, 0x0)\r\n    6505784   /usr/local/go/src/encoding/json/encode.go:690 +0x11b\r\n    6505785 encoding/json.(*arrayEncoder).(encoding/json.encode)-fm(0xc8db7ff970, 0x1ede780, 0xc914313ba0, 0x197, 0x0)\r\n    6505786   /usr/local/go/src/encoding/json/encode.go:697 +0x51\r\n    6505787 encoding/json.(*sliceEncoder).encode(0xc820686e10, 0xc8db7ff970, 0x1ede780, 0xc914313ba0, 0x197, 0x0)\r\n    6505788   /usr/local/go/src/encoding/json/encode.go:667 +0xb0\r\n    6505789 encoding/json.(*sliceEncoder).(encoding/json.encode)-fm(0xc8db7ff970, 0x1ede780, 0xc914313ba0, 0x197, 0x0)\r\n    6505790   /usr/local/go/src/encoding/json/encode.go:676 +0x51\r\n    6505791 encoding/json.(*structEncoder).encode(0xc82112f9b0, 0xc8db7ff970, 0x284d180, 0xc914313b60, 0x199, 0x7f6800000000)\r\n    6505792   /usr/local/go/src/encoding/json/encode.go:587 +0x2c4\r\n    6505793 encoding/json.(*structEncoder).(encoding/json.encode)-fm(0xc8db7ff970, 0x284d180, 0xc914313b60, 0x199, 0xc914313b00)\r\n    6505794   /usr/local/go/src/encoding/json/encode.go:601 +0x51\r\n    6505795 encoding/json.(*ptrEncoder).encode(0xc820686e18, 0xc8db7ff970, 0x2ab5400, 0xc914313b60, 0x16, 0x0)\r\n    6505796   /usr/local/go/src/encoding/json/encode.go:709 +0xea\r\n    6505797 encoding/json.(*ptrEncoder).(encoding/json.encode)-fm(0xc8db7ff970, 0x2ab5400, 0xc914313b60, 0x16, 0xc914313b00)\r\n    6505798   /usr/local/go/src/encoding/json/encode.go:714 +0x51\r\n    6505799 encoding/json.(*encodeState).reflectValue(0xc8db7ff970, 0x2ab5400, 0xc914313b60, 0x16)\r\n    6505800   /usr/local/go/src/encoding/json/encode.go:301 +0x6b\r\n    6505801 encoding/json.(*encodeState).marshal(0xc8db7ff970, 0x2ab5400, 0xc914313b60, 0x0, 0x0)\r\n    6505802   /usr/local/go/src/encoding/json/encode.go:274 +0xa9\r\n    6505803 encoding/json.(*Encoder).Encode(0xc8e39c2b58, 0x2ab5400, 0xc914313b60, 0x0, 0x0)\r\n    6505804   /usr/local/go/src/encoding/json/stream.go:188 +0x94\r\n    6505805 k8s.io/kubernetes/pkg/runtime/serializer/json.(*Serializer).EncodeToStream(0xc8201c4380, 0x7f683733c048, 0xc914313b60, 0x7f6837362488, 0xc8442738c8, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n    6505806   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/runtime/serializer/json/json.go:185 +0x323\r\n    6505807 k8s.io/kubernetes/pkg/runtime.(*SerializerInfo).EncodeToStream(0xc927271da0, 0x7f683733c048, 0xc914313b60, 0x7f6837362488, 0xc8442738c8, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n    6505808   <autogenerated>:34 +0xbe\r\n    6505809 k8s.io/kubernetes/pkg/runtime/serializer/versioning.(*codec).EncodeToStream(0xc91c5a6460, 0x7f683733c048, 0xc914313b60, 0x7f6837362488, 0xc8442738c8, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n    6505810   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/runtime/serializer/versioning/versioning.go:277 +0xc38\r\n    6505811 k8s.io/kubernetes/pkg/apiserver.writeNegotiated(0x7f683733ffa8, 0xc8206ce150, 0x0, 0x0, 0x2b85300, 0x2, 0x7f6837362120, 0xc8442738c8, 0xc8da52eee0, 0xc8, ...)\r\n    6505812   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:443 +0x30e\r\n    6505813 k8s.io/kubernetes/pkg/apiserver.write(0xc8, 0x0, 0x0, 0x2b85300, 0x2, 0x7f683733ffa8, 0xc8206ce150, 0x7f683733b850, 0xc8e6d11f20, 0x7f6837362120, ...)\r\n    6505814   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:427 +0xb70\r\n    6505815 k8s.io/kubernetes/pkg/apiserver.ListResource.func1(0xc8d9ea8510, 0xc8e6d11bc0)\r\n    6505816   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:327 +0xeb3\r\n    6505817 k8s.io/kubernetes/pkg/apiserver/metrics.InstrumentRouteFunc.func1(0xc8d9ea8510, 0xc8e6d11bc0)\r\n    6505818   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:101 +0x25c\r\n    6505819 k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc8201dea20, 0x7f6837361f98, 0xc844273898, 0xc8da52eee0)\r\n    6505820   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:249 +0xf30\r\n    6505821 k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).(k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch)-fm(0x7f6837361f98, 0xc844273898, 0xc8da52ee        e0)\r\n    6505822   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:95 +0x3e\r\n    6505823 net/http.HandlerFunc.ServeHTTP(0xc820d49880, 0x7f6837361f98, 0xc844273898, 0xc8da52eee0)\r\n    6505824   /usr/local/go/src/net/http/server.go:1618 +0x3a\r\n    6505825 net/http.(*ServeMux).ServeHTTP(0xc820416c90, 0x7f6837361f98, 0xc844273898, 0xc8da52eee0)\r\n    6505826   /usr/local/go/src/net/http/server.go:1910 +0x17d\r\n    6505827 k8s.io/kubernetes/pkg/apiserver.WithAuthorizationCheck.func1(0x7f6837361f98, 0xc844273898, 0xc8da52eee0)\r\n    6505828   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:482 +0xf4\r\n    6505829 net/http.HandlerFunc.ServeHTTP(0xc82028a6c0, 0x7f6837361f98, 0xc844273898, 0xc8da52eee0)\r\n    6505830   /usr/local/go/src/net/http/server.go:1618 +0x3a\r\n    6505831 k8s.io/kubernetes/pkg/apiserver.WithImpersonation.func1(0x7f6837361f98, 0xc844273898, 0xc8da52eee0)\r\n    6505832   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:417 +0xfe\r\n    6505833 net/http.HandlerFunc.ServeHTTP(0xc82028a700, 0x7f6837361f98, 0xc844273898, 0xc8da52eee0)\r\n    6505834   /usr/local/go/src/net/http/server.go:1618 +0x3a\r\n    6505835 k8s.io/kubernetes/pkg/auth/handlers.NewRequestAuthenticator.func1(0x7f6837361f98, 0xc844273898, 0xc8da52eee0)\r\n    6505836   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/auth/handlers/handlers.go:47 +0x1c5\r\n    6505837 net/http.HandlerFunc.ServeHTTP(0xc8203ff4f0, 0x7f6837361f98, 0xc844273898, 0xc8da52eee0)\r\n    6505838   /usr/local/go/src/net/http/server.go:1618 +0x3a\r\n    6505839 k8s.io/kubernetes/pkg/api.NewRequestContextFilter.func1(0x7f6837361f98, 0xc844273898, 0xc8da52eee0)\r\n    6505840   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:101 +0x157\r\n    6505841 net/http.HandlerFunc.ServeHTTP(0xc8203bb880, 0x7f6837361f98, 0xc844273898, 0xc8da52eee0)\r\n    6505842   /usr/local/go/src/net/http/server.go:1618 +0x3a\r\n    6505843 k8s.io/kubernetes/pkg/api.NewRequestContextFilter.func1(0x7f6837361f98, 0xc844273898, 0xc8da52eee0)\r\n    6505844   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:101 +0x157\r\n    6505845 net/http.HandlerFunc.ServeHTTP(0xc8203bb8a0, 0x7f6837361f98, 0xc844273898, 0xc8da52eee0)\r\n    6505846   /usr/local/go/src/net/http/server.go:1618 +0x3a\r\n    6505847 k8s.io/kubernetes/pkg/apiserver.(*timeoutHandler).ServeHTTP.func1(0xc8206395e0, 0x7f6837361f58, 0xc844273898, 0xc8da52eee0, 0xc8e6d11b00)\r\n    6505848   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:191 +0x91\r\n    6505849 created by k8s.io/kubernetes/pkg/apiserver.(*timeoutHandler).ServeHTTP\r\n    6505850   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:193 +0x15b\r\n\r\n\r\nIt seems that we are using UnsafeConvertToVersion too optimistically (or doing something wrong elsewhere).\r\n\r\n@smarterclayton @lavalamp @kubernetes/sig-api-machinery \r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/220339685","html_url":"https://github.com/kubernetes/kubernetes/issues/25880#issuecomment-220339685","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/25880","id":220339685,"user":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"created_at":"2016-05-19T14:23:34Z","updated_at":"2016-05-19T14:23:34Z","body":"Logs coming from this run:\r\nhttps://pantheon.corp.google.com/storage/browser/kubernetes-jenkins/logs/kubernetes-kubemark-gce-scale/1161/?project=kubernetes-jenkins"}}