{"action":"created","issue":{"url":"https://api.github.com/repos/envoyproxy/envoy-perf/issues/32","repository_url":"https://api.github.com/repos/envoyproxy/envoy-perf","labels_url":"https://api.github.com/repos/envoyproxy/envoy-perf/issues/32/labels{/name}","comments_url":"https://api.github.com/repos/envoyproxy/envoy-perf/issues/32/comments","events_url":"https://api.github.com/repos/envoyproxy/envoy-perf/issues/32/events","html_url":"https://github.com/envoyproxy/envoy-perf/issues/32","id":403134126,"node_id":"MDU6SXNzdWU0MDMxMzQxMjY=","number":32,"title":"Nighthawk: track-for-future list","user":{"login":"oschaaf","id":2239556,"node_id":"MDQ6VXNlcjIyMzk1NTY=","avatar_url":"https://avatars1.githubusercontent.com/u/2239556?v=4","gravatar_id":"","url":"https://api.github.com/users/oschaaf","html_url":"https://github.com/oschaaf","followers_url":"https://api.github.com/users/oschaaf/followers","following_url":"https://api.github.com/users/oschaaf/following{/other_user}","gists_url":"https://api.github.com/users/oschaaf/gists{/gist_id}","starred_url":"https://api.github.com/users/oschaaf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oschaaf/subscriptions","organizations_url":"https://api.github.com/users/oschaaf/orgs","repos_url":"https://api.github.com/users/oschaaf/repos","events_url":"https://api.github.com/users/oschaaf/events{/privacy}","received_events_url":"https://api.github.com/users/oschaaf/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2019-01-25T12:17:16Z","updated_at":"2019-04-15T22:06:53Z","closed_at":"2019-04-15T22:06:53Z","author_association":"CONTRIBUTOR","body":"This issue tracks high priority items that would bloat https://github.com/envoyproxy/envoy-perf/pull/31\r\n\r\n- This looks like a reasonable CLI to start with, but I would also like to take a proto variant of this as an input, and to make that canonical. [WIP]\r\n- ~~JSON and proto output of this data; this is probably easy to do and high priority, since most consumers will want this as an option. It's also easy to reformat to something human readable from JSON via a bunch of libraries/tools.~~\r\n- ~~gcovr test coverage. first attempts to get it working failed.~~\r\n- ~~Add content about benchmarking best practices [WIP nighthawk/README.md]~~\r\n- ~~Implement and use BenchmarkWorker~~\r\n- ~~Pull in and use C++ HdrHistogram library, get rid of the python stuff for that~~\r\n- ~~Currently we copied / hacked some stuff in ssl.h (e.g. MClientContextConfigImpl). I did this to avoid a cascade of dependencies because of one of the argument list of one of the interfaces involved. Revisit, evaluate, and see if we can get rid of this (or else discuss). [WIP]~~\r\n- ~~Evaluate and discuss usage of pending_requests in the Nighthawk client. In closed loop mode, we want to detect and warn if we hit this limit because latency measurements will be skewed.~~\r\n- Disable libevent threading support\r\n- ~~Overall, add docs where needed, and check for const correctness.~~\r\n- Harvey remarked: I assume this a is a per-worker rate limiter. We might want to have a global rate limit, since different workers might be moving at different effective rates, e.g. due to hyper-threading imbalance. This has to be weighed against potential lock contention, but I think we need something like this.\r\n- ~~Use the dispatcher on the worker thead to delay its start instead of calling usleep(). oschaaf: the dispatcher timer minimal resolution is 1ms, which is rather coarse for our purpose. I was thinking that maybe we can leverage the sequencer to guard the worker starting time, and spin to start workers with high precision timings.~~\r\n- Rate limiter: bursty behavior concerns due to thread scheduling. See if we can excite that behavior and address it. We may need to globally enforce rate limiting while doing this and introduce locks, so discussing the design of the rate limiter would be good.\r\n- check_format/fix_format in CI\r\n- ~~One thing we should decide on early is if we want an OS syscall abstraction layer. It's pretty nice for when you want to mock, and isn't _that_ much additional boilerplate to use. See https://github.com/envoyproxy/envoy/blob/master/include/envoy/api/os_sys_calls.h and its uses in Envoy.~~\r\n- ~~hdrhistogram_c is now included as a submodule. It would be better to make this an external repository in the Bazel sense and inject a BUILD file into it. See https://github.com/envoyproxy/envoy/blob/84d038f5a67a8dc8ca9753e9089a33d8110db3c2/api/bazel/repositories.bzl#L34 [WIP - https://github.com/envoyproxy/envoy-perf/pull/55]~~\r\n- [WIP] Add ASAN/TSAN tests to CI. Work in progress, experienced difficulties with getting this to work. However, we had to resort to quite a hack to override the linked that gets used. We want to revisit that and do it right once we figure out how.\r\n- Currently NH supports the simplest of cases only, sending GET requests. Wire through a to-be-designed (Http)RequestSynthesizer in the future to allow for more functionality.\r\n- We need to track response_trailers_ and look at grpc-status when doing gRPC load tests later on. It's fine not to do this for now, maybe just leave a TODO\r\n- ~~Redirect TCLAP logs to Envoy's logging subsystem. See  https://github.com/maddouri/tclap/blob/1e1cc4fb9abbc4bfcd62c73085c3c446fca681dd/include/tclap/CmdLineOutput.h~~\r\n- Do a pass to switch us to `using ::testing::XXX` in tests.\r\n- the stats implementation is converging rapidly towards what is already there in Envoy. We should look into reusing Envoy's functionality, and perhaps see if we can upstream HdrHistogram support and double check that the stdev implementation in Envoy is numerically stable \r\n- Factor out the boiler plate in `BenchmarkClientTest` to get a reusable integration test base.\r\n- Envoy has stats dumping protos, see https://github.com/envoyproxy/envoy/blob/master/api/envoy/admin/v2alpha/metrics.proto,  reuse them.\r\n- OS: discuss: Human usable interface & human readable output: currently we rely on tclap and fmt::format in c++ to interface with the benchmarking libs. Alternatively, we could use python or some such as a specialized tool to implement the frontend (e.g. cli, http server). \r\n- OS: discuss: Expose tls pool configuration options to control max requests allowed per connection, ciphers used, sigalg, etc. Discuss how we want to do that, do we directly accept Envoy's config for that? Or do we abstract and map our own config to Envoy's version? The first option is quick and easy, but uncontrolled. The latter may allow us to expose options more carefully and at our own pace.\r\n- OS: discuss: Snapshot the envoy/nh stats periodically from the workers for plotting them through time. What happens upon shutdown of the pool is interesting as well, so just pre and post shutdown is an interesting distinction too.\r\n- OS: discuss: Implement a series of tests in python that start up release builds of envoy + another server (e.g. nginx) with the same configuration, and run a predefined series of nighthawk tests against them. Adding this may also deprecate `client_test.cc` which does something hideous to fork the integration test server into its own process in the test framework.\r\n- OS: Add support for multiple connections in the h2 pool implementation.\r\n- OS: Controlling and separately managing connection management\r\n- OS: Separately track tls handshake timings, connection set up timings\r\n- Proactively prefetching connections that are lost while running a benchmark test would be a nice enhancement on top of the initial prefetching "},"comment":{"url":"https://api.github.com/repos/envoyproxy/envoy-perf/issues/comments/483436401","html_url":"https://github.com/envoyproxy/envoy-perf/issues/32#issuecomment-483436401","issue_url":"https://api.github.com/repos/envoyproxy/envoy-perf/issues/32","id":483436401,"node_id":"MDEyOklzc3VlQ29tbWVudDQ4MzQzNjQwMQ==","user":{"login":"oschaaf","id":2239556,"node_id":"MDQ6VXNlcjIyMzk1NTY=","avatar_url":"https://avatars1.githubusercontent.com/u/2239556?v=4","gravatar_id":"","url":"https://api.github.com/users/oschaaf","html_url":"https://github.com/oschaaf","followers_url":"https://api.github.com/users/oschaaf/followers","following_url":"https://api.github.com/users/oschaaf/following{/other_user}","gists_url":"https://api.github.com/users/oschaaf/gists{/gist_id}","starred_url":"https://api.github.com/users/oschaaf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oschaaf/subscriptions","organizations_url":"https://api.github.com/users/oschaaf/orgs","repos_url":"https://api.github.com/users/oschaaf/repos","events_url":"https://api.github.com/users/oschaaf/events{/privacy}","received_events_url":"https://api.github.com/users/oschaaf/received_events","type":"User","site_admin":false},"created_at":"2019-04-15T22:06:53Z","updated_at":"2019-04-15T22:06:53Z","author_association":"CONTRIBUTOR","body":"This moved to https://github.com/envoyproxy/nighthawk/issues/10, closing this here."}}
{"actor":{"display_login":"oschaaf"}}