{"action":"created","issue":{"url":"https://api.github.com/repos/Kong/kong/issues/4055","repository_url":"https://api.github.com/repos/Kong/kong","labels_url":"https://api.github.com/repos/Kong/kong/issues/4055/labels{/name}","comments_url":"https://api.github.com/repos/Kong/kong/issues/4055/comments","events_url":"https://api.github.com/repos/Kong/kong/issues/4055/events","html_url":"https://github.com/Kong/kong/issues/4055","id":387995562,"node_id":"MDU6SXNzdWUzODc5OTU1NjI=","number":4055,"title":"Kong throws 404 not found on route paths that exist at times.","user":{"login":"jeremyjpj0916","id":31913027,"node_id":"MDQ6VXNlcjMxOTEzMDI3","avatar_url":"https://avatars1.githubusercontent.com/u/31913027?v=4","gravatar_id":"","url":"https://api.github.com/users/jeremyjpj0916","html_url":"https://github.com/jeremyjpj0916","followers_url":"https://api.github.com/users/jeremyjpj0916/followers","following_url":"https://api.github.com/users/jeremyjpj0916/following{/other_user}","gists_url":"https://api.github.com/users/jeremyjpj0916/gists{/gist_id}","starred_url":"https://api.github.com/users/jeremyjpj0916/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeremyjpj0916/subscriptions","organizations_url":"https://api.github.com/users/jeremyjpj0916/orgs","repos_url":"https://api.github.com/users/jeremyjpj0916/repos","events_url":"https://api.github.com/users/jeremyjpj0916/events{/privacy}","received_events_url":"https://api.github.com/users/jeremyjpj0916/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2018-12-05T23:47:24Z","updated_at":"2018-12-12T02:21:05Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"### Summary\r\n\r\nWe have noticed in our Kong Gateway nodes times when common endpoints the gateway exposes throwing 404 route not found on a % of API calls. The specific proxy we focused on with this post( \"/F5/status\") does not route, has no auth, and serves as a ping up time endpoint that returns static 200 success. We do notice this as well on other endpoints that have auth and plugins as well, but the frequency in which our ping endpoint gets consumed is consistent and provides the best insight. \r\n\r\n### Steps To Reproduce\r\n\r\nReproducing consistently seems impossible from our perspective at this time, but we will elaborate with as much detail and screenshots as we can.\r\n\r\n1. Create combination of services and resources 1 to 1. We see 130 service route pairs. 300 plugins. Some global, some applied directly to routes(acl/auth).\r\n\r\n2. Add additional services + routes pairs with standard plugins over extended time. \r\n3. Suspicion is eventually the 404's will reveal themselves, and we have a high degree of confidence it does not happen globally across all worker processes.\r\n\r\nNew arbitrary Service+Route was made on the Gateway, specific important to note is created timestamp:\r\n![servicecreatedattimestamp](https://user-images.githubusercontent.com/31913027/49549744-aa05a380-f8b7-11e8-952c-df651c1fcbd6.png)\r\n\r\nConverted to UTC:\r\n![timestampconverted](https://user-images.githubusercontent.com/31913027/49549792-c7d30880-f8b7-11e8-913c-ec6aab2e89ea.png)\r\n\r\nAbove time matches identically with when the existing \"/F5/status\" began throwing 404 route not found errors:\r\n![f5statuserrorrateoccurs](https://user-images.githubusercontent.com/31913027/49549810-dd483280-f8b7-11e8-81d4-8282011797cc.png)\r\n\r\nYou can see a direct correlation to when that new service+route pair was created to when the gateway began to throw 404 not found errors on a route that exists and previously had no problems. Note the \"/F5/status\" endpoint takes consistent traffic at all time from health check monitors. \r\n\r\nInteresting bit its the % of errors to this individual Kong Node, we run 6 worker processes and the error rate % is almost perfect for 1 worker process showing impact:\r\n![f5status404rate](https://user-images.githubusercontent.com/31913027/49549847-f5b84d00-f8b7-11e8-9bc7-f4aa99cbe716.png)\r\n\r\n\r\nTo discuss our architecture we run Kong with Cassandra 3.X in 2 datacenters, 1 Kong node per data center. We run a 6 node Cassandra cluster, 3 C* nodes per datacenter. The errors only occurred in a single datacenter on the Kong node in our examples above, but both datacenters share identical settings. We redeploy Kong on a weekly basis every Monday early AM, but this error presented above Started on a Wednesday so we can't correlate the problem to any sort of Kong startup issue.\r\n\r\nTo us the behavior points to cache rebuilding during new resource creation based on what we can correlate. Sadly nothing in Kong logging catches anything of interest when we notice issue presenting itself. Also note it does not happen every time obviously so its a very hard issue to nail down.\r\n\r\nWe also notice the issue correcting itself too at times, we have not traced the correction to anything specific just yet, but I assume very likely its when further services and routes are created after the errors are occurring and what seems to be a problematic worker process has its router cleaned up again. \r\n\r\nOther points I can make are that production has not seen this issue with identical Kong configurations and architecture. But production has fewer proxies and has new services or routes added at a much lower frequency(1-2 per week vs 20+ in non-prod). \r\n\r\nI wonder at this time if it may be also safer for us to switch cache TTL back from the 0 infinity value to some arbitrary number of hours to force cycle on the resources. I suppose if it is indeed the cache as we suspect that that would actually make the frequency of this issue more prevalent possibly though. \r\n\r\nI may write a Kong health script that just arbitrarily grabs all routes on the gateway and calls each 1 one by 1 to ensure they don't return a 404 as a sanity check to run daily too right now. My biggest fear is as production grows in size and/or higher frequency in services/routes created daily we may begin to see the issue present itself there as well and that would cause dramatic impact to existing priority services if they start to 404 respond due to Kong not recognizing the proxy route exists in the db and caching appropriately.\r\n\r\nSorry I could not provide a 100% reproducible scenario for this situation, can only go off the analytics we have. Although if it yields some underlying bug in how Kong currently manages services and routes, that would bring huge stability to the core product.\r\n\r\n### Additional Details & Logs\r\n\r\n- Kong version 0.14.1\r\n- Kong error logs - \r\nKong Error logs reveal nothing about the 404 not found's from Kong's perspetive. Nothing gets logged during these events in terms of normal or debug execution.\r\n\r\n- Kong configuration (the output of a GET request to Kong's Admin port - see\r\n  https://docs.konghq.com/latest/admin-api/#retrieve-node-information)\r\n\r\n```\r\n{\r\n  \"plugins\": {\r\n    \"enabled_in_cluster\": [\r\n      \"correlation-id\",\r\n      \"acl\",\r\n      \"kong-oidc-multi-idp\",\r\n      \"oauth2\",\r\n      \"rate-limiting\",\r\n      \"cors\",\r\n      \"jwt\",\r\n      \"kong-spec-expose\",\r\n      \"request-size-limiting\",\r\n      \"kong-response-size-limiting\",\r\n      \"request-transformer\",\r\n      \"request-termination\",\r\n      \"kong-error-log\",\r\n      \"kong-oidc-implicit-token\",\r\n      \"kong-splunk-log\",\r\n      \"kong-upstream-jwt\",\r\n      \"kong-cluster-drain\",\r\n      \"statsd\"\r\n    ],\r\n    \"available_on_server\": {\r\n      \"kong-path-based-routing\": true,\r\n      \"kong-spec-expose\": true,\r\n      \"kong-cluster-drain\": true,\r\n      \"correlation-id\": true,\r\n      \"statsd\": true,\r\n      \"jwt\": true,\r\n      \"cors\": true,\r\n      \"kong-oidc-multi-idp\": true,\r\n      \"kong-response-size-limiting\": true,\r\n      \"kong-oidc-auth\": true,\r\n      \"kong-upstream-jwt\": true,\r\n      \"kong-error-log\": true,\r\n      \"request-termination\": true,\r\n      \"request-size-limiting\": true,\r\n      \"acl\": true,\r\n      \"rate-limiting\": true,\r\n      \"kong-service-virtualization\": true,\r\n      \"request-transformer\": true,\r\n      \"kong-oidc-implicit-token\": true,\r\n      \"kong-splunk-log\": true,\r\n      \"oauth2\": true\r\n    }\r\n  },\r\n  \"tagline\": \"Welcome to kong\",\r\n  \"configuration\": {\r\n    \"plugins\": [\r\n      \"kong-error-log\",\r\n      \"kong-oidc-implicit-token\",\r\n      \"kong-response-size-limiting\",\r\n      \"cors\",\r\n      \"request-transformer\",\r\n      \"kong-service-virtualization\",\r\n      \"kong-cluster-drain\",\r\n      \"kong-upstream-jwt\",\r\n      \"kong-splunk-log\",\r\n      \"kong-spec-expose\",\r\n      \"kong-oidc-auth\",\r\n      \"kong-path-based-routing\",\r\n      \"kong-oidc-multi-idp\",\r\n      \"correlation-id\",\r\n      \"oauth2\",\r\n      \"statsd\",\r\n      \"jwt\",\r\n      \"rate-limiting\",\r\n      \"acl\",\r\n      \"request-size-limiting\",\r\n      \"request-termination\"\r\n    ],\r\n    \"admin_ssl_enabled\": false,\r\n    \"lua_ssl_verify_depth\": 3,\r\n    \"trusted_ips\": [\r\n      \"0.0.0.0/0\",\r\n      \"::/0\"\r\n    ],\r\n    \"lua_ssl_trusted_certificate\": \"/usr/local/kong/ssl/kongcert.pem\",\r\n    \"loaded_plugins\": {\r\n      \"kong-path-based-routing\": true,\r\n      \"kong-spec-expose\": true,\r\n      \"kong-cluster-drain\": true,\r\n      \"correlation-id\": true,\r\n      \"statsd\": true,\r\n      \"jwt\": true,\r\n      \"cors\": true,\r\n      \"rate-limiting\": true,\r\n      \"kong-response-size-limiting\": true,\r\n      \"kong-oidc-auth\": true,\r\n      \"kong-upstream-jwt\": true,\r\n      \"acl\": true,\r\n      \"oauth2\": true,\r\n      \"kong-splunk-log\": true,\r\n      \"kong-oidc-implicit-token\": true,\r\n      \"kong-error-log\": true,\r\n      \"kong-service-virtualization\": true,\r\n      \"request-transformer\": true,\r\n      \"kong-oidc-multi-idp\": true,\r\n      \"request-size-limiting\": true,\r\n      \"request-termination\": true\r\n    },\r\n    \"cassandra_username\": \"****\",\r\n    \"admin_ssl_cert_csr_default\": \"/usr/local/kong/ssl/admin-kong-default.csr\",\r\n    \"ssl_cert_key\": \"/usr/local/kong/ssl/kongprivatekey.key\",\r\n    \"dns_resolver\": {},\r\n    \"pg_user\": \"kong\",\r\n    \"mem_cache_size\": \"1024m\",\r\n    \"cassandra_data_centers\": [\r\n      \"dc1:2\",\r\n      \"dc2:3\"\r\n    ],\r\n    \"nginx_admin_directives\": {},\r\n    \"cassandra_password\": \"******\",\r\n    \"custom_plugins\": {},\r\n    \"pg_host\": \"127.0.0.1\",\r\n    \"nginx_acc_logs\": \"/usr/local/kong/logs/access.log\",\r\n    \"proxy_listen\": [\r\n      \"0.0.0.0:8000\",\r\n      \"0.0.0.0:8443 ssl http2\"\r\n    ],\r\n    \"client_ssl_cert_default\": \"/usr/local/kong/ssl/kong-default.crt\",\r\n    \"ssl_cert_key_default\": \"/usr/local/kong/ssl/kong-default.key\",\r\n    \"dns_no_sync\": false,\r\n    \"db_update_propagation\": 5,\r\n    \"nginx_err_logs\": \"/usr/local/kong/logs/error.log\",\r\n    \"cassandra_port\": 9042,\r\n    \"dns_order\": [\r\n      \"LAST\",\r\n      \"SRV\",\r\n      \"A\",\r\n      \"CNAME\"\r\n    ],\r\n    \"dns_error_ttl\": 1,\r\n    \"headers\": [\r\n      \"latency_tokens\"\r\n    ],\r\n    \"dns_stale_ttl\": 4,\r\n    \"nginx_optimizations\": true,\r\n    \"database\": \"cassandra\",\r\n    \"pg_database\": \"kong\",\r\n    \"nginx_worker_processes\": \"auto\",\r\n    \"lua_package_cpath\": \"\",\r\n    \"admin_acc_logs\": \"/usr/local/kong/logs/admin_access.log\",\r\n    \"lua_package_path\": \"./?.lua;./?/init.lua;\",\r\n    \"nginx_pid\": \"/usr/local/kong/pids/nginx.pid\",\r\n    \"upstream_keepalive\": 120,\r\n    \"cassandra_contact_points\": [\r\n      \"server8429\",\r\n      \"server8431\",\r\n      \"server8432\",\r\n      \"server8433\",\r\n      \"server8445\",\r\n      \"server8428\"\r\n    ],\r\n    \"admin_access_log\": \"off\",\r\n    \"client_ssl_cert_csr_default\": \"/usr/local/kong/ssl/kong-default.csr\",\r\n    \"proxy_listeners\": [\r\n      {\r\n        \"ssl\": false,\r\n        \"ip\": \"0.0.0.0\",\r\n        \"proxy_protocol\": false,\r\n        \"port\": 8000,\r\n        \"http2\": false,\r\n        \"listener\": \"0.0.0.0:8000\"\r\n      },\r\n      {\r\n        \"ssl\": true,\r\n        \"ip\": \"0.0.0.0\",\r\n        \"proxy_protocol\": false,\r\n        \"port\": 8443,\r\n        \"http2\": true,\r\n        \"listener\": \"0.0.0.0:8443 ssl http2\"\r\n      }\r\n    ],\r\n    \"proxy_ssl_enabled\": true,\r\n    \"proxy_access_log\": \"off\",\r\n    \"ssl_cert_csr_default\": \"/usr/local/kong/ssl/kong-default.csr\",\r\n    \"enabled_headers\": {\r\n      \"latency_tokens\": true,\r\n      \"X-Upstream-Status\": false,\r\n      \"X-Proxy-Latency\": true,\r\n      \"server_tokens\": false,\r\n      \"Server\": false,\r\n      \"Via\": false,\r\n      \"X-Upstream-Latency\": true\r\n    },\r\n    \"cassandra_ssl\": true,\r\n    \"cassandra_local_datacenter\": \"DC2\",\r\n    \"db_resurrect_ttl\": 30,\r\n    \"db_update_frequency\": 5,\r\n    \"cassandra_consistency\": \"LOCAL_QUORUM\",\r\n    \"client_max_body_size\": \"100m\",\r\n    \"admin_error_log\": \"/dev/stderr\",\r\n    \"pg_ssl_verify\": false,\r\n    \"dns_not_found_ttl\": 30,\r\n    \"pg_ssl\": false,\r\n    \"cassandra_repl_factor\": 1,\r\n    \"cassandra_lb_policy\": \"RequestDCAwareRoundRobin\",\r\n    \"cassandra_repl_strategy\": \"SimpleStrategy\",\r\n    \"nginx_kong_conf\": \"/usr/local/kong/nginx-kong.conf\",\r\n    \"error_default_type\": \"text/plain\",\r\n    \"nginx_http_directives\": {},\r\n    \"real_ip_header\": \"X-Forwarded-For\",\r\n    \"kong_env\": \"/usr/local/kong/.kong_env\",\r\n    \"cassandra_schema_consensus_timeout\": 10000,\r\n    \"dns_hostsfile\": \"/etc/hosts\",\r\n    \"admin_listeners\": [\r\n      {\r\n        \"ssl\": false,\r\n        \"ip\": \"0.0.0.0\",\r\n        \"proxy_protocol\": false,\r\n        \"port\": 8001,\r\n        \"http2\": false,\r\n        \"listener\": \"0.0.0.0:8001\"\r\n      }\r\n    ],\r\n    \"ssl_ciphers\": \"ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256\",\r\n    \"ssl_cert\": \"/usr/local/kong/ssl/kongcert.crt\",\r\n    \"prefix\": \"/usr/local/kong\",\r\n    \"admin_ssl_cert_key_default\": \"/usr/local/kong/ssl/admin-kong-default.key\",\r\n    \"cassandra_ssl_verify\": true,\r\n    \"db_cache_ttl\": 0,\r\n    \"ssl_cipher_suite\": \"modern\",\r\n    \"real_ip_recursive\": \"on\",\r\n    \"proxy_error_log\": \"/dev/stderr\",\r\n    \"client_ssl_cert_key_default\": \"/usr/local/kong/ssl/kong-default.key\",\r\n    \"nginx_daemon\": \"off\",\r\n    \"anonymous_reports\": false,\r\n    \"cassandra_timeout\": 5000,\r\n    \"nginx_proxy_directives\": {},\r\n    \"pg_port\": 5432,\r\n    \"log_level\": \"debug\",\r\n    \"client_body_buffer_size\": \"50m\",\r\n    \"client_ssl\": false,\r\n    \"lua_socket_pool_size\": 30,\r\n    \"admin_ssl_cert_default\": \"/usr/local/kong/ssl/admin-kong-default.crt\",\r\n    \"cassandra_keyspace\": \"kong_stage\",\r\n    \"ssl_cert_default\": \"/usr/local/kong/ssl/kong-default.crt\",\r\n    \"nginx_conf\": \"/usr/local/kong/nginx.conf\",\r\n    \"admin_listen\": [\r\n      \"0.0.0.0:8001\"\r\n    ]\r\n  },\r\n  \"version\": \"0.14.1\",\r\n  \"node_id\": \"cf0c92a9-724d-4972-baed-599785cff5ed\",\r\n  \"lua_version\": \"LuaJIT 2.1.0-beta3\",\r\n  \"prng_seeds\": {\r\n    \"pid: 73\": 183184219419,\r\n    \"pid: 71\": 114224634222,\r\n    \"pid: 72\": 213242339120,\r\n    \"pid: 70\": 218221808514,\r\n    \"pid: 69\": 233240145991,\r\n    \"pid: 68\": 231238177547\r\n  },\r\n  \"timers\": {\r\n    \"pending\": 4,\r\n    \"running\": 0\r\n  },\r\n  \"hostname\": \"kong-62-hxqnq\"\r\n}\r\n```\r\n- Operating system: Kong's Docker Alpine version 3.6 in github repo"},"comment":{"url":"https://api.github.com/repos/Kong/kong/issues/comments/446436991","html_url":"https://github.com/Kong/kong/issues/4055#issuecomment-446436991","issue_url":"https://api.github.com/repos/Kong/kong/issues/4055","id":446436991,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NjQzNjk5MQ==","user":{"login":"thibaultcha","id":1125431,"node_id":"MDQ6VXNlcjExMjU0MzE=","avatar_url":"https://avatars2.githubusercontent.com/u/1125431?v=4","gravatar_id":"","url":"https://api.github.com/users/thibaultcha","html_url":"https://github.com/thibaultcha","followers_url":"https://api.github.com/users/thibaultcha/followers","following_url":"https://api.github.com/users/thibaultcha/following{/other_user}","gists_url":"https://api.github.com/users/thibaultcha/gists{/gist_id}","starred_url":"https://api.github.com/users/thibaultcha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thibaultcha/subscriptions","organizations_url":"https://api.github.com/users/thibaultcha/orgs","repos_url":"https://api.github.com/users/thibaultcha/repos","events_url":"https://api.github.com/users/thibaultcha/events{/privacy}","received_events_url":"https://api.github.com/users/thibaultcha/received_events","type":"User","site_admin":false},"created_at":"2018-12-12T02:21:05Z","updated_at":"2018-12-12T02:21:05Z","author_association":"MEMBER","body":"When I was trying to reproduce this in the plane earlier, I hit a couple of other issues with the DAO that I am in the process of fixing. Unfortunately, traveling, and other thhings are getting in my way these days.\r\n\r\nThose issues that I found should be mitigated by the router mutex though. \r\n\r\nThat said, I was not able to replicate your observations Jeremy. But as I am about to board another 12h flight, who knows, maybe I will have another chance at it, or @bungle will!\r\n\r\nThanks for giving rc4 a spin. "}}
{"actor":{"display_login":"thibaultcha"}}