<p>Issue also posted at:</p>

<ul>
<li><a href="https://github.com/istio/istio/issues/21195" rel="nofollow noreferrer">https://github.com/istio/istio/issues/21195</a></li>
<li><a href="https://www.reddit.com/r/istio/comments/f57v2a/help_failed_to_run_bookinfo_example_behind_proxy/" rel="nofollow noreferrer">https://www.reddit.com/r/istio/comments/f57v2a/help_failed_to_run_bookinfo_example_behind_proxy/</a></li>
</ul>

<p><strong>Problem:</strong></p>

<p>Following the bookinfo example, when trying to apply bookinfo-gateway, I get the following error:</p>

<pre><code>Error from server (InternalError): error when creating "samples/bookinfo/networking/bookinfo-gateway.yaml": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: context deadline exceeded
</code></pre>

<p><strong>Output:</strong></p>

<pre><code>$ kubectl --v=9 apply -f samples/bookinfo/networking/bookinfo-gateway.yaml
I0216 18:01:08.548290    4904 loader.go:375] Config loaded from file:  /home/user/.kube/config
I0216 18:01:08.550426    4904 round_trippers.go:423] curl -k -v -XGET  -H "User-Agent: kubectl/v1.17.3 (linux/amd64) kubernetes/06ad960" -H "Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf" 'https://ha-lb-ip:6443/openapi/v2?timeout=32s'
I0216 18:01:08.600310    4904 round_trippers.go:443] GET https://ha-lb-ip:6443/openapi/v2?timeout=32s 200 OK in 49 milliseconds
I0216 18:01:08.600348    4904 round_trippers.go:449] Response Headers:
I0216 18:01:08.600355    4904 round_trippers.go:452]     Accept-Ranges: bytes
I0216 18:01:08.600361    4904 round_trippers.go:452]     X-Varied-Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf
I0216 18:01:08.600366    4904 round_trippers.go:452]     Content-Type: application/octet-stream
I0216 18:01:08.600371    4904 round_trippers.go:452]     Etag: "DCA49D599C62F0A8DDF840BBF0F4DB11A2B0C9805F7F6CEB19F163F61CA7D40F9E7A3607B007A74CCD6DBA6565BE6E6E3085528F7FD18EDAE99BABE9702D8700"
I0216 18:01:08.600378    4904 round_trippers.go:452]     Last-Modified: Sun, 16 Feb 2020 11:54:46 GMT
I0216 18:01:08.600430    4904 round_trippers.go:452]     Vary: Accept-Encoding
I0216 18:01:08.600435    4904 round_trippers.go:452]     Vary: Accept
I0216 18:01:08.600439    4904 round_trippers.go:452]     Date: Sun, 16 Feb 2020 12:01:08 GMT
I0216 18:01:08.817775    4904 request.go:1015] Response Body:
00000000  0a 03 32 2e 30 12 15 0a  0a 4b 75 62 65 72 6e 65  |..2.0....Kuberne|
00000010  74 65 73 12 07 76 31 2e  31 37 2e 33 42 93 f7 a9  |tes..v1.17.3B...|
00000020  01 12 ae 27 0a 29 2f 61  70 69 2f 76 31 2f 77 61  |...'.)/api/v1/wa|
00000030  74 63 68 2f 6e 61 6d 65  73 70 61 63 65 73 2f 7b  |tch/namespaces/{|
00000040  6e 61 6d 65 73 70 61 63  65 7d 2f 70 6f 64 73 12  |namespace}/pods.|
00000050  80 27 12 97 04 0a 07 63  6f 72 65 5f 76 31 1a 6f  |.'.....core_v1.o|
00000060  77 61 74 63 68 20 69 6e  64 69 76 69 64 75 61 6c  |watch individual|
00000070  20 63 68 61 6e 67 65 73  20 74 6f 20 61 20 6c 69  | changes to a li|
00000080  73 74 20 6f 66 20 50 6f  64 2e 20 64 65 70 72 65  |st of Pod. depre|
00000090  63 61 74 65 64 3a 20 75  73 65 20 74 68 65 20 27  |cated: use the '|
000000a0  77 61 74 63 68 27 20 70  61 72 61 6d 65 74 65 72  |watch' parameter|
000000b0  20 77 69 74 68 20 61 20  6c 69 73 74 20 6f 70 65  | with a list ope|
000000c0  72 61 74 69 6f 6e 20 69  6e 73 74 65 61 64 2e 2a  |ration instead.*|
000000d0  1c 77 61 74 63 68 43 6f  72 65 56 31 4e 61 6d 65  |.watchCoreV1Name|
000000e0  73 70 61 63 65 64 50 6f  64 4c 69 73 74 32 10 61  |spacedPodList2.a|
000000f0  70 70 6c 69 63 61 74 69  6f 6e 2f 6a 73 6f 6e 32  |pplication/json2|
00000100  10 61 70 70 6c 69 63 61  74 69 6f 6e 2f 79 61 6d  |.application/yam|
00000110  6c 32 23 61 70 70 6c 69  63 61 74 69 6f 6e 2f 76  |l2#application/v|
00000120  6e 64 2e 6b 75 62 65 72  6e 65 74 65 73 2e 70 72  |nd.kubernetes.pr|
00000130  6f 74 6f 62 75 66 32 1d  61 70 70 6c 69 63 61 74  |otobuf2.applicat|
00000140  69 6f 6e 2f 6a 73 6f 6e  3b 73 74 72 65 61 6d 3d  |ion/json;stream=|
00000150  77 61 74 63 68 32 30 61  70 70 6c 69 63 61 74 69  |watch20applicati|
00000160  6f 6e 2f 76 6e 64 2e 6b  75 62 65 72 6e 65 74 65  |on/vnd.kubernete|
00000170  73 2e 70 72 6f 74 6f 62  75 66 3b 73 74 72 65 61  |s.protobuf;strea|
00000180  6d 3d 77 61 74 63 68 3a  03 2a 2f 2a 4a 6b 0a 50  |m=watch:.*/*Jk.P|
00000190  0a 03 32 30 30 12 49 0a  47 0a 02 4f 4b 12 41 0a  |..200.I.G..OK.A.|
000001a0  3f 0a 3d 23 2f 64 65 66  69 6e 69 74 69 6f 6e 73  |?.=#/definitions|
000001b0  2f 69 6f 2e 6b 38 73 2e  61 70 69 6d 61 63 68 69  |/io.k8s.apimachi|
000001c0  6e 65 72 79 2e 70 6b 67  2e 61 70 69 73 2e 6d 65  |nery.pkg.apis.me|
000001d0  74 61 2e 76 31 2e 57 61  74 63 68 45 76 65 6e 74  |ta.v1.WatchEvent|
000001e0  0a 17 0a 03 34 30 31 12  10 0a 0e 0a 0c 55 6e 61  |....401......Una|
000001f0  75 74 68 6f 72 69 7a 65  64 52 05 68 74 74 70 73  |uthorizedR.https|
00000200  6a 23 0a 13 78 2d 6b 75  62 65 72 6e 65 74 65 73  |j#..x-kubernetes|
00000210  2d 61 63 74 69 6f 6e 12  0c 12 0a 77 61 74 63 68  |-action....watch|
00000220  6c 69 73 74 0a 6a 45 0a  1f 78 2d 6b 75 62 65 72  |list.jE..x-kuber|
00000230  6e 65 74 65 73 2d 67 72  6f 75 70 2d 76 65 72 73  |netes-group-vers|
00000240  69 6f 6e 2d 6b 69 6e 64  12 22 12 20 6b 69 6e 64  |ion-kind.". kind|
00000250  3a 20 50 6f 64 0a 76 65  72 73 69 6f 6e 3a 20 76  |: Pod.version: v|
00000260  31 0a 67 72 6f 75 70 3a  20 22 22 0a 4a 82 04 0a  |1.group: "".J...|
00000270  ff 03 12 fc 03 1a f9 03  12 05 71 75 65 72 79 1a  |..........query.|
00000280  ce 03 61 6c 6c 6f 77 57  61 74 63 68 42 6f 6f 6b  |..allowWatchBook|
00000290  6d 61 72 6b 73 20 72 65  71 75 65 73 74 73 20 77  |marks requests w|
000002a0  61 74 63 68 20 65 76 65  6e 74 73 20 77 69 74 68  |atch events with|
000002b0  20 74 79 70 65 20 22 42  4f 4f 4b 4d 41 52 4b 22  | type "BOOKMARK"|
000002c0  2e 20 53 65 72 76 65 72  73 20 74 68 61 74 20 64  |. Servers that d|
000002d0  6f 20 6e 6f 74 20 69 6d  70 6c 65 6d 65 6e 74 20  |o not implement |
000002e0  62 6f 6f 6b 6d 61 72 6b  73 20 6d 61 79 20 69 67  |bookmarks may ig|
000002f0  6e 6f 72 65 20 74 68 69  73 20 66 6c 61 67 20 61  |nore this flag a|
00000300  6e 64 20 62 6f 6f 6b 6d  61 72 6b 73 20 61 72 65  |nd bookmarks are|
00000310  20 73 65 6e 74 20 61 74  20 74 68 65 20 73 65 72  | sent at the ser|
00000320  76 65 72 27 73 20 64 69  73 63 72 65 74 69 6f 6e  |ver's discretion|
00000330  2e 20 43 6c 69 65 6e 74  73 20 73 68 6f 75 6c 64  |. Clients should|
00000340  20 6e 6f 74 20 61 73 73  75 6d 65 20 62 6f 6f 6b  | not assume book|
00000350  6d 61 72 6b 73 20 61 72  65 20 72 65 74 75 72 6e  |marks are return|
00000360  65 64 20 61 74 20 61 6e  79 20 73 70 65 63 69 66  |ed at any specif|
00000370  69 63 20 69 6e 74 65 72  76 61 6c 2c 20 6e 6f 72  |ic interval, nor|
00000380  20 6d 61 79 20 74 68 65  79 20 61 73 73 75 6d 65  | may they assume|
00000390  20 74 68 65 20 73 65 72  76 65 72 20 77 69 6c 6c  | the server will|
000003a0  20 73 65 6e 64 20 61 6e  79 20 42 4f 4f 4b 4d 41  | send any BOOKMA|
000003b0  52 4b 20 65 76 65 6e 74  20 64 75 72 69 6e 67 20  |RK event during |
000003c0  61 20 73 65 73 73 69 6f  6e 2e 20 49 66 20 74 68  |a session. If th|
000003d0  69 73 20 69 73 20 6e 6f  74 20 61 20 77 61 74 63  |is is not a watc|
000003e0  68 2c 20 74 68 69 73 20  66 69 65 6c 64 20 69 73  |h, this field is|
000003f0  20 69 67 6e 6f 72 65 64  2e 20 49 66 20 74 68 65  | ignored. If the|
00000400  20 66 65 61 74 75 72 65  20 67 61 74 65 20 57 61  | feature gate Wa|
00000410  74 63 68 42 6f 6f 6b 6d  61 72 6b 73 20 69 73 20  |tchBookmarks is |
00000420  6e 6f 74 20 65 6e 61 62  6c 65 64 20 69 6e 20 61  |not enabled in a|
00000430  70 69 73 65 72 76 65 72  2c 20 74 68 69 73 20 66  |piserver, this f|
00000440  69 65 6c 64 20 69 73 20  69 67 6e 6f 72 65 64 2e  |ield is ignored.|
00000450  22 13 61 6c 6c 6f 77 57  61 74 63 68 42 6f 6f 6b  |".allowWatchBook|
00000460  6d 61 72 6b 73 32 07 62  6f 6f 6c 65 61 6e a0 01  |marks2.boolean..|
00000470  01 4a ef 09 0a ec 09 12  e9 09 1a e6 09 12 05 71  |.J.............q|
00000480  75 65 72 79 1a c7 09 54  68 65 20 63 6f 6e 74 69  |uery...The conti|
00000490  6e 75 65 20 6f 70 74 69  6f 6e 20 73 68 6f 75 6c  |nue option shoul|
000004a0  64 20 62 65 20 73 65 74  20 77 68 65 6e 20 72 65  |d be set when re|
000004b0  74 72 69 65 76 69 6e 67  20 6d 6f 72 65 20 72 65  |trieving more re|
000004c0  73 75 6c 74 73 20 66 72  6f 6d 20 74 68 65 20 73  |sults from the s|
000004d0  65 72 76 65 72 2e 20 53  69 6e 63 65 20 74 68 69  |erver. Since thi|
000004e0  73 20 76 61 6c 75 65 20  69 73 20 73 65 72 76 65  |s value is serve|
000004f0  72 20 64 65 66 69 6e 65  64 2c 20 63 6c 69 65 6e  |r defined, clien|
00000500  74 73 20 6d 61 79 20 6f  6e 6c 79 20 75 73 65 20  |ts may only use |
00000510  74 68 65 20 63 6f 6e 74  69 6e 75 65 20 76 61 6c  |the continue val|
00000520  75 65 20 66 72 6f 6d 20  61 20 70 72 65 76 69 6f  |ue from a previo|
00000530  75 73 20 71 75 65 72 79  20 72 65 73 75 6c 74 20  |us query result |
00000540  77 69 74 68 20 69 64 65  6e 74 69 63 61 6c 20 71  |with identical q|
00000550  75 65 72 79 20 70 61 72  61 6d 65 74 65 72 73 20  |uery parameters |
00000560  28 65 78 63 65 70 74 20  66 6f 72 20 74 68 65 20  |(except for the |
00000570  76 61 6c 75 65 20 6f 66  20 63 6f 6e 74 69 6e 75  |value of continu|
00000580  65 29 20 61 6e 64 20 74  68 65 20 73 65 72 76 65  |e) and the serve|
00000590  72 20 6d 61 79 20 72 65  6a 65 63 74 20 61 20 63  |r may reject a c|
000005a0  6f 6e 74 69 6e 75 65 20  76 61 6c 75 65 20 69 74  |ontinue value it|
000005b0  20 64 6f 65 73 20 6e 6f  74 20 72 65 63 6f 67 6e  | does not recogn|
000005c0  69 7a 65 2e 20 49 66 20  74 68 65 20 73 70 65 63  |ize. If the spec|
000005d0  69 66 69 65 64 20 63 6f  6e 74 69 6e 75 65 20 76  |ified continue v|
000005e0  61 6c 75 65 20 69 73 20  6e 6f 20 6c 6f 6e 67 65  |alue is no longe|
000005f0  72 20 76 61 6c 69 64 20  77 68 65 74 68 65 72 20  |r valid whether |
00000600  64 75 65 20 74 6f 20 65  78 70 69 72 61 74 69 6f  |due to expiratio|
00000610  6e 20 28 67 65 6e 65 72  61 6c 6c 79 20 66 69 76  |n (generally fiv|
00000620  65 20 74 6f 20 66 69 66  74 65 65 6e 20 6d 69 6e  |e to fifteen min|
00000630  75 74 65 73 29 20 6f 72  20 61 20 63 6f 6e 66 69  |utes) or a confi|
00000640  67 75 72 61 74 69 6f 6e  20 63 68 61 6e 67 65 20  |guration change |
00000650  6f 6e 20 74 68 65 20 73  65 72 76 65 72 2c 20 74  |on the server, t|
00000660  68 65 20 73 65 72 76 65  72 20 77 69 6c 6c 20 72  |he server will r|
00000670  65 73 70 6f 6e 64 20 77  69 74 68 20 61 20 34 31  |espond with a 41|
00000680  30 20 52 65 73 6f 75 72  63 65 45 78 70 69 72 65  |0 ResourceExpire|
00000690  64 20 65 72 72 6f 72 20  74 6f 67 65 74 68 65 72  |d error together|
000006a0  20 77 69 74 68 20 61 20  63 6f 6e 74 69 6e 75 65  | with a continue|
000006b0  20 74 6f 6b 65 6e 2e 20  49 66 20 74 68 65 20 63  | token. If the c|
000006c0  6c 69 65 6e 74 20 6e 65  65 64 73 20 61 20 63 6f  |lient needs a co|
000006d0  6e 73 69 73 74 65 6e 74  20 6c 69 73 74 2c 20 69  |nsistent list, i|
000006e0  74 20 6d 75 73 74 20 72  65 73 74 61 72 74 20 74  |t must restart t|
000006f0  68 65 69 72 20 6c 69 73  74 20 77 69 74 68 6f 75  |heir list withou|
00000700  74 20 74 68 65 20 63 6f  6e 74 69 6e 75 65 20 66  |t the continue f|
00000710  69 65 6c 64 2e 20 4f 74  68 65 72 77 69 73 65 2c  |ield. Otherwise,|
00000720  20 74 68 65 20 63 6c 69  65 6e 74 20 6d 61 79 20  | the client may |
00000730  73 65 6e 64 20 61 6e 6f  74 68 65 72 20 6c 69 73  |send another lis|
00000740  74 20 72 65 71 75 65 73  74 20 77 69 74 68 20 74  |t request with t|
00000750  68 65 20 74 6f 6b 65 6e  20 72 65 63 65 69 76 65  |he token receive|
00000760  64 20 77 69 74 68 20 74  68 65 20 34 31 30 20 65  |d with the 410 e|
00000770  72 72 6f 72 2c 20 74 68  65 20 73 65 72 76 65 72  |rror, the server|
00000780  20 77 69 6c 6c 20 72 65  73 70 6f 6e 64 20 77 69  | will respond wi|
00000790  74 68 20 61 20 6c 69 73  74 20 73 74 61 72 74 69  |th a list starti|
000007a0  6e 67 20 66 72 6f 6d 20  74 68 65 20 6e 65 78 74  |ng from the next|
000007b0  20 6b 65 79 2c 20 62 75  74 20 66 72 6f 6d 20 74  | key, but from t|
000007c0  68 65 20 6c 61 74 65 73  74 20 73 6e 61 70 73 68  |he latest snapsh|
000007d0  6f 74 2c 20 77 68 69 63  68 20 69 73 20 69 6e 63  |ot, which is inc|
000007e0  6f 6e 73 69 73 74 65 6e  74 20 66 72 6f 6d 20 74  |onsistent from t|
000007f0  68 65 20 70 72 65 76 69  6f 75 73 20 6c 69 73 74  |he previous list|
00000800  20 72 65 73 75 6c 74 73  20 2d 20 6f 62 6a 65 63  | results - objec|
00000810  74 73 20 74 68 61 74 20  61 72 65 20 63 [truncated 17345571 chars]
I0216 18:01:08.944681    4904 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/v1.17.3 (linux/amd64) kubernetes/06ad960" 'https://ha-lb-ip:6443/apis/networking.istio.io/v1alpha3/namespaces/default/gateways/bookinfo-gateway'
I0216 18:01:08.988784    4904 round_trippers.go:443] GET https://ha-lb-ip:6443/apis/networking.istio.io/v1alpha3/namespaces/default/gateways/bookinfo-gateway 404 Not Found in 44 milliseconds
I0216 18:01:08.988834    4904 round_trippers.go:449] Response Headers:
I0216 18:01:08.988840    4904 round_trippers.go:452]     Content-Length: 258
I0216 18:01:08.988844    4904 round_trippers.go:452]     Date: Sun, 16 Feb 2020 12:01:08 GMT
I0216 18:01:08.988848    4904 round_trippers.go:452]     Content-Type: application/json
I0216 18:01:08.988896    4904 request.go:1017] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"gateways.networking.istio.io \"bookinfo-gateway\" not found","reason":"NotFound","details":{"name":"bookinfo-gateway","group":"networking.istio.io","kind":"gateways"},"code":404}
I0216 18:01:08.989774    4904 request.go:1017] Request Body: {"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.istio.io/v1alpha3\",\"kind\":\"Gateway\",\"metadata\":{\"annotations\":{},\"name\":\"bookinfo-gateway\",\"namespace\":\"default\"},\"spec\":{\"selector\":{\"istio\":\"ingressgateway\"},\"servers\":[{\"hosts\":[\"*\"],\"port\":{\"name\":\"http\",\"number\":80,\"protocol\":\"HTTP\"}}]}}\n"},"name":"bookinfo-gateway","namespace":"default"},"spec":{"selector":{"istio":"ingressgateway"},"servers":[{"hosts":["*"],"port":{"name":"http","number":80,"protocol":"HTTP"}}]}}
I0216 18:01:08.989839    4904 round_trippers.go:423] curl -k -v -XPOST  -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.17.3 (linux/amd64) kubernetes/06ad960" 'https://ha-lb-ip:6443/apis/networking.istio.io/v1alpha3/namespaces/default/gateways'
I0216 18:01:38.996165    4904 round_trippers.go:443] POST https://ha-lb-ip:6443/apis/networking.istio.io/v1alpha3/namespaces/default/gateways 500 Internal Server Error in 30006 milliseconds
I0216 18:01:38.996302    4904 round_trippers.go:449] Response Headers:
I0216 18:01:38.996315    4904 round_trippers.go:452]     Content-Type: application/json
I0216 18:01:38.996320    4904 round_trippers.go:452]     Content-Length: 481
I0216 18:01:38.996349    4904 round_trippers.go:452]     Date: Sun, 16 Feb 2020 12:01:38 GMT
I0216 18:01:38.996563    4904 request.go:1017] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: context deadline exceeded","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: context deadline exceeded"}]},"code":500}
I0216 18:01:38.999383    4904 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/v1.17.3 (linux/amd64) kubernetes/06ad960" 'https://ha-lb-ip:6443/apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices/bookinfo'
I0216 18:01:39.042269    4904 round_trippers.go:443] GET https://ha-lb-ip:6443/apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices/bookinfo 404 Not Found in 42 milliseconds
I0216 18:01:39.042304    4904 round_trippers.go:449] Response Headers:
I0216 18:01:39.042310    4904 round_trippers.go:452]     Content-Type: application/json
I0216 18:01:39.042315    4904 round_trippers.go:452]     Content-Length: 256
I0216 18:01:39.042319    4904 round_trippers.go:452]     Date: Sun, 16 Feb 2020 12:01:39 GMT
I0216 18:01:39.042352    4904 request.go:1017] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"virtualservices.networking.istio.io \"bookinfo\" not found","reason":"NotFound","details":{"name":"bookinfo","group":"networking.istio.io","kind":"virtualservices"},"code":404}
I0216 18:01:39.043083    4904 request.go:1017] Request Body: {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.istio.io/v1alpha3\",\"kind\":\"VirtualService\",\"metadata\":{\"annotations\":{},\"name\":\"bookinfo\",\"namespace\":\"default\"},\"spec\":{\"gateways\":[\"bookinfo-gateway\"],\"hosts\":[\"*\"],\"http\":[{\"match\":[{\"uri\":{\"exact\":\"/productpage\"}},{\"uri\":{\"prefix\":\"/static\"}},{\"uri\":{\"exact\":\"/login\"}},{\"uri\":{\"exact\":\"/logout\"}},{\"uri\":{\"prefix\":\"/api/v1/products\"}}],\"route\":[{\"destination\":{\"host\":\"productpage\",\"port\":{\"number\":9080}}}]}]}}\n"},"name":"bookinfo","namespace":"default"},"spec":{"gateways":["bookinfo-gateway"],"hosts":["*"],"http":[{"match":[{"uri":{"exact":"/productpage"}},{"uri":{"prefix":"/static"}},{"uri":{"exact":"/login"}},{"uri":{"exact":"/logout"}},{"uri":{"prefix":"/api/v1/products"}}],"route":[{"destination":{"host":"productpage","port":{"number":9080}}}]}]}}
I0216 18:01:39.043172    4904 round_trippers.go:423] curl -k -v -XPOST  -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.17.3 (linux/amd64) kubernetes/06ad960" 'https://ha-lb-ip:6443/apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices'
I0216 18:02:09.049842    4904 round_trippers.go:443] POST https://ha-lb-ip:6443/apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices 500 Internal Server Error in 30006 milliseconds
I0216 18:02:09.050031    4904 round_trippers.go:449] Response Headers:
I0216 18:02:09.050043    4904 round_trippers.go:452]     Content-Type: application/json
I0216 18:02:09.050052    4904 round_trippers.go:452]     Content-Length: 481
I0216 18:02:09.050059    4904 round_trippers.go:452]     Date: Sun, 16 Feb 2020 12:02:09 GMT
I0216 18:02:09.050249    4904 request.go:1017] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: context deadline exceeded","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: context deadline exceeded"}]},"code":500}
I0216 18:02:09.051955    4904 helpers.go:203] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error when creating \"samples/bookinfo/networking/bookinfo-gateway.yaml\": Internal error occurred: failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: context deadline exceeded",
  "reason": "InternalError",
  "details": {
    "causes": [
      {
        "message": "failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: context deadline exceeded"
      }
    ]
  },
  "code": 500
}]
I0216 18:02:09.052104    4904 helpers.go:203] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error when creating \"samples/bookinfo/networking/bookinfo-gateway.yaml\": Internal error occurred: failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: context deadline exceeded",
  "reason": "InternalError",
  "details": {
    "causes": [
      {
        "message": "failed calling webhook \"pilot.validation.istio.io\": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: context deadline exceeded"
      }
    ]
  },
  "code": 500
}]
F0216 18:02:09.052210    4904 helpers.go:114] Error from server (InternalError): error when creating "samples/bookinfo/networking/bookinfo-gateway.yaml": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: context deadline exceeded
Error from server (InternalError): error when creating "samples/bookinfo/networking/bookinfo-gateway.yaml": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: context deadline exceeded
</code></pre>

<p><strong>Related Information:</strong></p>

<ul>
<li>Docker version 19.03.5</li>
<li>Kubernetes version 1.17.3</li>
<li>Istio version 1.4.4</li>
<li>Kubernetes clusters are run behind company's proxy server. CNI using <code>kube-flannel</code></li>
<li>The <code>docker.service.d</code> config has proxy configured at <code>/etc/systemd/system/docker.service.d/proxy.conf</code>.</li>
<li><code>NO_PROXY</code>/<code>no_proxy</code> is set to <code>localhost,127.0.0.1,::1,.grameenphone.com,10.10.18.188,10.10.23.57,10.10.23.58,10.10.23.59,10.10.23.60,10.10.23.61,kubernetes.default,.validation.istio.io,.istio.io,.istio-system.svc,.svc,.istio-system,.svc.cluster.local,.cluster.local,10.244.0.0/16</code>.</li>
<li>Installed ISTIO using <code>istioctl</code>, yaml given below:</li>
</ul>

<pre><code>apiVersion: install.istio.io/v1alpha2
kind: IstioControlPlane
spec:
  defaultNamespace: istio-system
  cni:
    enabled: true
  gateways:
    components:
      egressGateway:
        enabled: false
      ingressGateway:
        enabled: true
    enabled: true
  values:
    cni:
      excludeNamespaces:
        - istio-system
        - kube-system
    gateways:
      istio-ingressgateway:
        type: NodePort
    global:
      configValidation: true
    grafana:
      enabled: true
    kiali:
      enabled: true
    tracing:
      enabled: true
</code></pre>

<ul>
<li>Istio install was successful, verified using g<code>isitoctl verify-install</code>.</li>
<li>Tried modifying <code>/etc/kubernetes/manifests/kube-apiserver.yaml</code> by adding <code>env:</code> section to the container, but still fails to create <code>bookinfo-gateway</code>.</li>
</ul>

<pre><code>    env:
      - name: http_proxy
        value: http://10.10.20.107:3828
      - name: https_proxy
        value: http://10.10.20.107:3828
      - name: no_proxy
        value: localhost,127.0.0.1,::1,.grameenphone.com,10.10.18.188,10.10.23.57,10.10.23.58,10.10.23.59,10.10.23.60,10.10.23.61,kubernetes.default,.validation.istio.io,.istio.io,.istio-system.svc,.svc,.istio-system,.svc.cluster.local,.cluster.local,10.244.0.0/16,10.96.0.0/12
</code></pre>

<ul>
<li>For sidecar injection, I'm following the manual procedure.</li>
<li>All Isito pods are up &amp; running.</li>
<li>Log from <code>kube-apiserver</code>:</li>
</ul>

<pre><code>I0217 06:36:08.719672       1 controller.go:606] quota admission added evaluator for: deployments.apps
I0217 06:37:19.151894       1 trace.go:116] Trace[2116455659]: "Call validating webhook" configuration:istio-galley,webhook:pilot.validation.istio.io,resource:networking.istio.io/v1alpha3, Resource=gateways,subresource:,operation:CREATE,UID:de57f49e-fd19-44ea-99d7-414dfec0981f (started: 2020-02-17 06:36:49.150893916 +0000 UTC m=+6172.698922922) (total time: 30.000884409s):
Trace[2116455659]: [30.000884409s] [30.000884409s] END
W0217 06:37:19.151963       1 dispatcher.go:133] Failed calling webhook, failing closed pilot.validation.istio.io: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: context deadline exceeded
I0217 06:37:19.152376       1 trace.go:116] Trace[2129026394]: "Create" url:/apis/networking.istio.io/v1alpha3/namespaces/default/gateways,user-agent:kubectl/v1.17.3 (linux/amd64) kubernetes/06ad960,client:10.10.18.188 (started: 2020-02-17 06:36:49.149255324 +0000 UTC m=+6172.697284349) (total time: 30.003096239s):
Trace[2129026394]: [30.003096239s] [30.002534955s] END
I0217 06:37:49.167492       1 trace.go:116] Trace[1063136940]: "Call validating webhook" configuration:istio-galley,webhook:pilot.validation.istio.io,resource:networking.istio.io/v1alpha3, Resource=virtualservices,subresource:,operation:CREATE,UID:2dd4b3e7-8333-4c1d-8222-dd53f8ce2db4 (started: 2020-02-17 06:37:19.166772885 +0000 UTC m=+6202.714801862) (total time: 30.000661809s):
Trace[1063136940]: [30.000661809s] [30.000661809s] END
W0217 06:37:49.167530       1 dispatcher.go:133] Failed calling webhook, failing closed pilot.validation.istio.io: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: context deadline exceeded
I0217 06:37:49.167996       1 trace.go:116] Trace[639287810]: "Create" url:/apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices,user-agent:kubectl/v1.17.3 (linux/amd64) kubernetes/06ad960,client:10.10.18.188 (started: 2020-02-17 06:37:19.165649222 +0000 UTC m=+6202.713678242) (total time: 30.002325938s):
Trace[639287810]: [30.002325938s] [30.001822425s] END
</code></pre>

<p>I googled/looked into other related issues &amp; tried their solutions, but did not worked.</p>

<p>If there is any additional information required, please do let me know.</p>

<p>Seems that <a href="https://istio.io/docs/reference/commands/galley/" rel="nofollow noreferrer"><code>galley</code></a> stopped or doesn't answer.</p>

<blockquote>
  <p>Galley provides configuration management services for Istio.</p>
</blockquote>

<p>So, pls check these two steps: </p>

<h1>1. Is <a href="https://istio.io/docs/reference/commands/galley/" rel="nofollow noreferrer"><code>galley</code></a> running?</h1>

<p><code>galley probe</code></p>

<h1>2. If it runs, try the WA</h1>

<p>Due to this known issue <a href="https://github.com/istio/istio/issues/17162" rel="nofollow noreferrer">istio/istio#17162</a>, seems that your problem is related to api-server proxy settings.</p>

<p>And there is a <a href="https://github.com/kubeflow/manifests/issues/422#issuecomment-566336889" rel="nofollow noreferrer">workaround</a> </p>

<p>Try to use <code>--set global.configValidation=false</code> when installing <code>istio</code></p>

<p>See also <a href="https://istio.io/docs/ops/setup/validation/" rel="nofollow noreferrer">manual on <code>istio</code> validation</a> </p>
