{"action":"created","issue":{"url":"https://api.github.com/repos/TykTechnologies/tyk/issues/1701","repository_url":"https://api.github.com/repos/TykTechnologies/tyk","labels_url":"https://api.github.com/repos/TykTechnologies/tyk/issues/1701/labels{/name}","comments_url":"https://api.github.com/repos/TykTechnologies/tyk/issues/1701/comments","events_url":"https://api.github.com/repos/TykTechnologies/tyk/issues/1701/events","html_url":"https://github.com/TykTechnologies/tyk/pull/1701","id":323484671,"number":1701,"title":"added changes to do less i/o to Redis","user":{"login":"dencoded","id":33698537,"avatar_url":"https://avatars1.githubusercontent.com/u/33698537?v=4","gravatar_id":"","url":"https://api.github.com/users/dencoded","html_url":"https://github.com/dencoded","followers_url":"https://api.github.com/users/dencoded/followers","following_url":"https://api.github.com/users/dencoded/following{/other_user}","gists_url":"https://api.github.com/users/dencoded/gists{/gist_id}","starred_url":"https://api.github.com/users/dencoded/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dencoded/subscriptions","organizations_url":"https://api.github.com/users/dencoded/orgs","repos_url":"https://api.github.com/users/dencoded/repos","events_url":"https://api.github.com/users/dencoded/events{/privacy}","received_events_url":"https://api.github.com/users/dencoded/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-05-16T06:29:46Z","updated_at":"2018-05-19T00:29:41Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/TykTechnologies/tyk/pulls/1701","html_url":"https://github.com/TykTechnologies/tyk/pull/1701","diff_url":"https://github.com/TykTechnologies/tyk/pull/1701.diff","patch_url":"https://github.com/TykTechnologies/tyk/pull/1701.patch"},"body":"While looking at performance I've noticed that we speak to redis too often than it is supposed to do. I tried to put break point and found that https://github.com/TykTechnologies/tyk/blob/v2.6.1/user/session.go#L96 almost never returning `false` so we always updating session in redis\r\n\r\nAdded some changes I believe should make speak to Redis much less:\r\n\r\n- in `ApplyPolicies` we set Tags to zero value which was different when initial state was nil and session hash was different as well\r\n- made `Session.FirstSeenHash` not to be public so no way it can appear in msgpack and be used to calculate new hash - https://github.com/TykTechnologies/tyk/blob/v2.6.1/user/session.go#L87 was always giving new data and it made hash to be different\r\n- call `SetFirstSeenHash()` when we have real update for session in redis so next call to `UpdateSession` won't touch redis (if session hasn't changed)\r\n- delete key from SessionCache when session was updated in Redis\r\n- something not related - looks like we were calculating session hash in a wrong way, we had hasher but never wrote data there and calculated it, call of method `Sum(encoded)` is just giving current hash (empty as never calculated) appended to passed array, so it was always returning just the same `encoded` var value\r\n\r\nsome benchmarking vs. master:\r\n```\r\nbenchmark                                                   old ns/op     new ns/op     delta\r\nBenchmarkURLReplacer-8                                      18445         18449         +0.02%\r\nBenchmarkTagHeaders-8                                       974           1072          +10.06%\r\nBenchmarkDefaultVersion-8                                   1607676       1106262       -31.19%\r\nBenchmarkGetVersionFromRequest/Header_location-8            497267        498422        +0.23%\r\nBenchmarkGetVersionFromRequest/URL_param_location-8         483260        502072        +3.89%\r\nBenchmarkGetVersionFromRequest/URL_location-8               483978        477496        -1.34%\r\nBenchmarkApiReload-8                                        3900897       3607856       -7.51%\r\nBenchmarkMultiSession_BA_Standard_OK-8                      1577178       2382575       +51.07%\r\nBenchmarkBearerTokenAuthKeySession-8                        1748503       1915178       +9.53%\r\nBenchmarkMultiAuthBackwardsCompatibleSession-8              1765264       1939514       +9.87%\r\nBenchmarkBasicAuth-8                                        160339388     159637538     -0.44%\r\nBenchmarkContextVarsMiddleware-8                            1360221       1373033       +0.94%\r\nBenchmarkMiddlewareContextVars_ProcessRequest_cookies-8     139025        139807        +0.56%\r\nBenchmarkHMACAuthSessionPass-8                              1516810       1247421       -17.76%\r\nBenchmarkHMACAuthSessionPassWithHeaderField-8               1558462       1418480       -8.98%\r\nBenchmarkIPBlacklistMiddleware-8                            28280         29024         +2.63%\r\nBenchmarkIPMiddlewarePass-8                                 19031         18774         -1.35%\r\nBenchmarkJWTSessionHMAC-8                                   657598        389382        -40.79%\r\nBenchmarkJWTSessionRSA-8                                    774483        507916        -34.42%\r\nBenchmarkJWTSessionRSABearer-8                              796075        499417        -37.27%\r\nBenchmarkJWTSessionRSAWithRawSourceOnWithClientID-8         720965        507177        -29.65%\r\nBenchmarkJWTSessionRSAWithRawSource-8                       713440        509946        -28.52%\r\nBenchmarkJWTSessionRSAWithJWK-8                             713125        503046        -29.46%\r\nBenchmarkJWTSessionRSAWithEncodedJWK-8                      712967        510735        -28.36%\r\nBenchmarkProcessRequestLiveQuotaLimit-8                     600988        610759        +1.63%\r\nBenchmarkProcessRequestOffThreadQuotaLimit-8                466115        458657        -1.60%\r\nBenchmarkProcessRequestLiveRedisRollingLimiter-8            966534        898430        -7.05%\r\nBenchmarkProcessRequestOffThreadRedisRollingLimiter-8       759907        742076        -2.35%\r\nBenchmarkStripAuth_stripFromHeaders-8                       8860          8848          -0.14%\r\nBenchmarkStripAuth_stripFromParams-8                        39959         40636         +1.69%\r\nBenchmarkTransformNonAscii-8                                15489         15487         -0.01%\r\nBenchmarkTransformJSONMarshal-8                             16552         16241         -1.88%\r\nBenchmarkRewriter-8                                         197181        200454        +1.66%\r\nBenchmarkValidateJSONSchema-8                               1834339       1820883       -0.73%\r\nBenchmarkVersioning-8                                       2349581       2066859       -12.03%\r\nBenchmarkVirtualEndpoint-8                                  353343        351438        -0.54%\r\nBenchmarkApplyPolicies-8                                    26759         26631         -0.48%\r\nBenchmarkResponseHeaderInjection-8                          2202799       2207808       +0.23%\r\nBenchmarkCopyRequestResponse-8                              15302         15290         -0.08%\r\n\r\nbenchmark                                                   old allocs     new allocs     delta\r\nBenchmarkURLReplacer-8                                      44             44             +0.00%\r\nBenchmarkTagHeaders-8                                       15             15             +0.00%\r\nBenchmarkDefaultVersion-8                                   1782           1694           -4.94%\r\nBenchmarkApiReload-8                                        23907          23907          +0.00%\r\nBenchmarkMultiSession_BA_Standard_OK-8                      464            290            -37.50%\r\nBenchmarkBearerTokenAuthKeySession-8                        387            245            -36.69%\r\nBenchmarkMultiAuthBackwardsCompatibleSession-8              391            247            -36.83%\r\nBenchmarkBasicAuth-8                                        25994          25939          -0.21%\r\nBenchmarkContextVarsMiddleware-8                            2009           2009           +0.00%\r\nBenchmarkMiddlewareContextVars_ProcessRequest_cookies-8     28             28             +0.00%\r\nBenchmarkHMACAuthSessionPass-8                              388            293            -24.48%\r\nBenchmarkHMACAuthSessionPassWithHeaderField-8               415            383            -7.71%\r\nBenchmarkIPBlacklistMiddleware-8                            148            148            +0.00%\r\nBenchmarkIPMiddlewarePass-8                                 94             94             +0.00%\r\nBenchmarkJWTSessionHMAC-8                                   707            618            -12.59%\r\nBenchmarkJWTSessionRSA-8                                    782            693            -11.38%\r\nBenchmarkJWTSessionRSABearer-8                              784            694            -11.48%\r\nBenchmarkJWTSessionRSAWithRawSourceOnWithClientID-8         807            710            -12.02%\r\nBenchmarkJWTSessionRSAWithRawSource-8                       791            702            -11.25%\r\nBenchmarkJWTSessionRSAWithJWK-8                             795            706            -11.19%\r\nBenchmarkJWTSessionRSAWithEncodedJWK-8                      799            710            -11.14%\r\nBenchmarkProcessRequestLiveQuotaLimit-8                     572            585            +2.27%\r\nBenchmarkProcessRequestOffThreadQuotaLimit-8                534            547            +2.43%\r\nBenchmarkProcessRequestLiveRedisRollingLimiter-8            2095           2141           +2.20%\r\nBenchmarkProcessRequestOffThreadRedisRollingLimiter-8       2321           2337           +0.69%\r\nBenchmarkStripAuth_stripFromHeaders-8                       66             66             +0.00%\r\nBenchmarkStripAuth_stripFromParams-8                        220            220            +0.00%\r\nBenchmarkTransformNonAscii-8                                98             98             +0.00%\r\nBenchmarkTransformJSONMarshal-8                             101            101            +0.00%\r\nBenchmarkRewriter-8                                         626            626            +0.00%\r\nBenchmarkValidateJSONSchema-8                               2588           2587           -0.04%\r\nBenchmarkVersioning-8                                       3042           2975           -2.20%\r\nBenchmarkVirtualEndpoint-8                                  384            384            +0.00%\r\nBenchmarkApplyPolicies-8                                    102            102            +0.00%\r\nBenchmarkResponseHeaderInjection-8                          2720           2719           -0.04%\r\nBenchmarkCopyRequestResponse-8                              76             76             +0.00%\r\n\r\nbenchmark                                                   old bytes     new bytes     delta\r\nBenchmarkURLReplacer-8                                      849           849           +0.00%\r\nBenchmarkTagHeaders-8                                       384           384           +0.00%\r\nBenchmarkDefaultVersion-8                                   251629        225030        -10.57%\r\nBenchmarkApiReload-8                                        1789668       1789641       -0.00%\r\nBenchmarkMultiSession_BA_Standard_OK-8                      90636         27633         -69.51%\r\nBenchmarkBearerTokenAuthKeySession-8                        79509         22634         -71.53%\r\nBenchmarkMultiAuthBackwardsCompatibleSession-8              80672         23099         -71.37%\r\nBenchmarkBasicAuth-8                                        1341352       1322977       -1.37%\r\nBenchmarkContextVarsMiddleware-8                            308360        308362        +0.00%\r\nBenchmarkMiddlewareContextVars_ProcessRequest_cookies-8     2386          2386          +0.00%\r\nBenchmarkHMACAuthSessionPass-8                              79035         47557         -39.83%\r\nBenchmarkHMACAuthSessionPassWithHeaderField-8               81027         68852         -15.03%\r\nBenchmarkIPBlacklistMiddleware-8                            32279         32273         -0.02%\r\nBenchmarkIPMiddlewarePass-8                                 26720         26720         +0.00%\r\nBenchmarkJWTSessionHMAC-8                                   107345        87272         -18.70%\r\nBenchmarkJWTSessionRSA-8                                    136960        107958        -21.18%\r\nBenchmarkJWTSessionRSABearer-8                              138167        109118        -21.02%\r\nBenchmarkJWTSessionRSAWithRawSourceOnWithClientID-8         128578        109372        -14.94%\r\nBenchmarkJWTSessionRSAWithRawSource-8                       122819        107191        -12.72%\r\nBenchmarkJWTSessionRSAWithJWK-8                             122863        107254        -12.70%\r\nBenchmarkJWTSessionRSAWithEncodedJWK-8                      123042        107397        -12.72%\r\nBenchmarkProcessRequestLiveQuotaLimit-8                     88770         86782         -2.24%\r\nBenchmarkProcessRequestOffThreadQuotaLimit-8                86994         85011         -2.28%\r\nBenchmarkProcessRequestLiveRedisRollingLimiter-8            149933        145331        -3.07%\r\nBenchmarkProcessRequestOffThreadRedisRollingLimiter-8       158852        153182        -3.57%\r\nBenchmarkStripAuth_stripFromHeaders-8                       2800          2800          +0.00%\r\nBenchmarkStripAuth_stripFromParams-8                        14161         14161         +0.00%\r\nBenchmarkTransformNonAscii-8                                11060         11060         +0.00%\r\nBenchmarkTransformJSONMarshal-8                             11508         11508         +0.00%\r\nBenchmarkRewriter-8                                         405130        405130        +0.00%\r\nBenchmarkValidateJSONSchema-8                               291872        291537        -0.11%\r\nBenchmarkVersioning-8                                       318004        297837        -6.34%\r\nBenchmarkVirtualEndpoint-8                                  70864         70866         +0.00%\r\nBenchmarkApplyPolicies-8                                    12585         12585         +0.00%\r\nBenchmarkResponseHeaderInjection-8                          528933        528867        -0.01%\r\nBenchmarkCopyRequestResponse-8                              35584         35584         +0.00%\r\n```\r\n\r\nstill need to figure out what happened with BenchmarkMultiSession_BA_Standard_OK"},"comment":{"url":"https://api.github.com/repos/TykTechnologies/tyk/issues/comments/390364507","html_url":"https://github.com/TykTechnologies/tyk/pull/1701#issuecomment-390364507","issue_url":"https://api.github.com/repos/TykTechnologies/tyk/issues/1701","id":390364507,"user":{"login":"matiasinsaurralde","id":20110,"avatar_url":"https://avatars1.githubusercontent.com/u/20110?v=4","gravatar_id":"","url":"https://api.github.com/users/matiasinsaurralde","html_url":"https://github.com/matiasinsaurralde","followers_url":"https://api.github.com/users/matiasinsaurralde/followers","following_url":"https://api.github.com/users/matiasinsaurralde/following{/other_user}","gists_url":"https://api.github.com/users/matiasinsaurralde/gists{/gist_id}","starred_url":"https://api.github.com/users/matiasinsaurralde/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matiasinsaurralde/subscriptions","organizations_url":"https://api.github.com/users/matiasinsaurralde/orgs","repos_url":"https://api.github.com/users/matiasinsaurralde/repos","events_url":"https://api.github.com/users/matiasinsaurralde/events{/privacy}","received_events_url":"https://api.github.com/users/matiasinsaurralde/received_events","type":"User","site_admin":false},"created_at":"2018-05-19T00:29:41Z","updated_at":"2018-05-19T00:29:41Z","author_association":"MEMBER","body":"Seems like `BenchmarkMultiSession_BA_Standard_OK` is impacted by the new murmur calls?"}}
{"actor":{"display_login":"matiasinsaurralde"}}