{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes-sigs/aws-alb-ingress-controller/issues/584","repository_url":"https://api.github.com/repos/kubernetes-sigs/aws-alb-ingress-controller","labels_url":"https://api.github.com/repos/kubernetes-sigs/aws-alb-ingress-controller/issues/584/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes-sigs/aws-alb-ingress-controller/issues/584/comments","events_url":"https://api.github.com/repos/kubernetes-sigs/aws-alb-ingress-controller/issues/584/events","html_url":"https://github.com/kubernetes-sigs/aws-alb-ingress-controller/pull/584","id":356964785,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEzMDk0ODg0","number":584,"title":"*: add AWS API call latency histogram","user":{"login":"gyuho","id":6799218,"node_id":"MDQ6VXNlcjY3OTkyMTg=","avatar_url":"https://avatars0.githubusercontent.com/u/6799218?v=4","gravatar_id":"","url":"https://api.github.com/users/gyuho","html_url":"https://github.com/gyuho","followers_url":"https://api.github.com/users/gyuho/followers","following_url":"https://api.github.com/users/gyuho/following{/other_user}","gists_url":"https://api.github.com/users/gyuho/gists{/gist_id}","starred_url":"https://api.github.com/users/gyuho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gyuho/subscriptions","organizations_url":"https://api.github.com/users/gyuho/orgs","repos_url":"https://api.github.com/users/gyuho/repos","events_url":"https://api.github.com/users/gyuho/events{/privacy}","received_events_url":"https://api.github.com/users/gyuho/received_events","type":"User","site_admin":false},"labels":[{"id":959251099,"node_id":"MDU6TGFiZWw5NTkyNTEwOTk=","url":"https://api.github.com/repos/kubernetes-sigs/aws-alb-ingress-controller/labels/cncf-cla:%20yes","name":"cncf-cla: yes","color":"bfe5bf","default":false},{"id":959251913,"node_id":"MDU6TGFiZWw5NTkyNTE5MTM=","url":"https://api.github.com/repos/kubernetes-sigs/aws-alb-ingress-controller/labels/size/XL","name":"size/XL","color":"ee5500","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-09-04T20:31:03Z","updated_at":"2018-09-07T18:27:58Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/kubernetes-sigs/aws-alb-ingress-controller/pulls/584","html_url":"https://github.com/kubernetes-sigs/aws-alb-ingress-controller/pull/584","diff_url":"https://github.com/kubernetes-sigs/aws-alb-ingress-controller/pull/584.diff","patch_url":"https://github.com/kubernetes-sigs/aws-alb-ingress-controller/pull/584.patch"},"body":"Add `ObserveAPIRequest` method to track AWS resource create/modify/delete API calls.\r\n\r\n### Why?\r\n\r\nLatency distribution of ALB Rule modification is useful for debugging slow Ingress rule updates, DescribeRules API call latency can explain why “kubectl describe ing” takes long time to return when there are more than 30 rules, slow CreateSecurityGroup calls may cause slow bootstrapping process, etc.\r\n\r\n### Example output\r\n\r\n```\r\ncurl -L ${URL}/metrics\r\n\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_bucket{operation=\"CreateRule\",le=\"0.0128\"} 0\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_bucket{operation=\"CreateRule\",le=\"0.0256\"} 27\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_bucket{operation=\"CreateRule\",le=\"0.0512\"} 39\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_bucket{operation=\"CreateRule\",le=\"0.1024\"} 42\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_bucket{operation=\"CreateRule\",le=\"0.2048\"} 43\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_bucket{operation=\"CreateRule\",le=\"0.4096\"} 43\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_bucket{operation=\"CreateRule\",le=\"0.8192\"} 58\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_bucket{operation=\"CreateRule\",le=\"1.6384\"} 70\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_bucket{operation=\"CreateRule\",le=\"3.2768\"} 70\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_bucket{operation=\"CreateRule\",le=\"6.5536\"} 70\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_bucket{operation=\"CreateRule\",le=\"13.1072\"} 70\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_bucket{operation=\"CreateRule\",le=\"+Inf\"} 70\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_sum{operation=\"CreateRule\"} 23.170110464\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_count{operation=\"CreateRule\"} 70\r\naws_alb_ingress_controller_aws_api_requests_latency_seconds_bucket{operation=\"CreateTargetGroup\",le=\"0.0001\"} 0\r\n```\r\n\r\nNote that those histogram numbers are cumulative, where `le` means less of equal. The delta is the number of requests for the corresponding range.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes-sigs/aws-alb-ingress-controller/issues/comments/419526613","html_url":"https://github.com/kubernetes-sigs/aws-alb-ingress-controller/pull/584#issuecomment-419526613","issue_url":"https://api.github.com/repos/kubernetes-sigs/aws-alb-ingress-controller/issues/584","id":419526613,"node_id":"MDEyOklzc3VlQ29tbWVudDQxOTUyNjYxMw==","user":{"login":"bigkraig","id":508403,"node_id":"MDQ6VXNlcjUwODQwMw==","avatar_url":"https://avatars1.githubusercontent.com/u/508403?v=4","gravatar_id":"","url":"https://api.github.com/users/bigkraig","html_url":"https://github.com/bigkraig","followers_url":"https://api.github.com/users/bigkraig/followers","following_url":"https://api.github.com/users/bigkraig/following{/other_user}","gists_url":"https://api.github.com/users/bigkraig/gists{/gist_id}","starred_url":"https://api.github.com/users/bigkraig/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bigkraig/subscriptions","organizations_url":"https://api.github.com/users/bigkraig/orgs","repos_url":"https://api.github.com/users/bigkraig/repos","events_url":"https://api.github.com/users/bigkraig/events{/privacy}","received_events_url":"https://api.github.com/users/bigkraig/received_events","type":"User","site_admin":false},"created_at":"2018-09-07T18:27:58Z","updated_at":"2018-09-07T18:27:58Z","author_association":"COLLABORATOR","body":"Here is a way of getting the timing data with some session handlers:\r\n\r\n```\r\npackage main\r\n\r\nimport (\r\n\t\"context\"\r\n\t\"fmt\"\r\n\t\"net/http/httptrace\"\r\n\t\"time\"\r\n\r\n\t\"github.com/aws/aws-sdk-go/aws/request\"\r\n\t\"github.com/aws/aws-sdk-go/aws/session\"\r\n\t\"github.com/aws/aws-sdk-go/service/ec2\"\r\n)\r\n\r\ntype ContextKeytype int\r\n\r\nvar ContextKey = new(ContextKeytype)\r\n\r\ntype timeData struct {\r\n\tconnStart time.Time\r\n}\r\n\r\nvar prometheusSignHandler = request.NamedHandler{\r\n\tName: \"PrometheusSignHandler\",\r\n\tFn: func(r *request.Request) {\r\n\t\ttrace := &httptrace.ClientTrace{\r\n\t\t\tConnectStart: func(network, addr string) {\r\n\t\t\t\ttd := r.HTTPRequest.Context().Value(ContextKey).(*timeData)\r\n\t\t\t\ttd.connStart = time.Now()\r\n\t\t\t},\r\n\t\t}\r\n\r\n\t\tr.HTTPRequest = r.HTTPRequest.WithContext(context.WithValue(r.HTTPRequest.Context(), ContextKey, &timeData{}))\r\n\t\tr.HTTPRequest = r.HTTPRequest.WithContext(httptrace.WithClientTrace(r.HTTPRequest.Context(), trace))\r\n\t},\r\n}\r\n\r\nvar prometheusCompleteHandler = request.NamedHandler{\r\n\tName: \"PrometheusCompleteHandler\",\r\n\tFn: func(r *request.Request) {\r\n\t\ttiming := r.HTTPRequest.Context().Value(ContextKey).(*timeData)\r\n\t\tfmt.Printf(\"service: %s\\noperation: %s\\nduration: %v\\n\",\r\n\t\t\tr.ClientInfo.ServiceName,\r\n\t\t\tr.Operation.Name,\r\n\t\t\ttime.Now().Sub(timing.connStart),\r\n\t\t)\r\n\t},\r\n}\r\n\r\nfunc main() {\r\n\tsess := session.Must(session.NewSession())\r\n\r\n\tsess.Handlers.Sign.PushFrontNamed(prometheusSignHandler)\r\n\tsess.Handlers.Complete.PushFrontNamed(prometheusCompleteHandler)\r\n\r\n\tsvc := ec2.New(sess)\r\n\tsvc.DescribeAccountAttributes(&ec2.DescribeAccountAttributesInput{})\r\n}\r\n```"}}
{"actor":{"display_login":"bigkraig"}}