{"action":"created","issue":{"url":"https://api.github.com/repos/OpenConext/OpenConext-deploy/issues/48","repository_url":"https://api.github.com/repos/OpenConext/OpenConext-deploy","labels_url":"https://api.github.com/repos/OpenConext/OpenConext-deploy/issues/48/labels{/name}","comments_url":"https://api.github.com/repos/OpenConext/OpenConext-deploy/issues/48/comments","events_url":"https://api.github.com/repos/OpenConext/OpenConext-deploy/issues/48/events","html_url":"https://github.com/OpenConext/OpenConext-deploy/issues/48","id":138622279,"number":48,"title":"Running ./provision-agent fails on aa","user":{"login":"precurse","id":2087223,"avatar_url":"https://avatars.githubusercontent.com/u/2087223?v=3","gravatar_id":"","url":"https://api.github.com/users/precurse","html_url":"https://github.com/precurse","followers_url":"https://api.github.com/users/precurse/followers","following_url":"https://api.github.com/users/precurse/following{/other_user}","gists_url":"https://api.github.com/users/precurse/gists{/gist_id}","starred_url":"https://api.github.com/users/precurse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/precurse/subscriptions","organizations_url":"https://api.github.com/users/precurse/orgs","repos_url":"https://api.github.com/users/precurse/repos","events_url":"https://api.github.com/users/precurse/events{/privacy}","received_events_url":"https://api.github.com/users/precurse/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-03-05T00:04:25Z","updated_at":"2016-03-07T07:57:17Z","closed_at":null,"body":"Hello,\r\n\r\nI've been trying to get a test environment going using the current code base, but keep getting the following error when I run it.\r\n```\r\nTASK [aa : wait for aa to start] ***********************************************\r\nfatal: [192.168.66.79]: FAILED! => {\"changed\": false, \"elapsed\": 300, \"failed\": true, \"msg\": \"Timeout when waiting for 127.0.0.1:9198\"}\r\n\r\n```\r\nAccessing https://vm.openconext.org/ gives:\r\n```\r\n503 Service Unavailable\r\n\r\nNo server is available to handle this request.\r\n```\r\n\r\nI've poked around to see if I can figure it out, but haven't been able to yet..  Any help would be great!\r\n\r\nThanks,\r\nAndrew\r\n\r\nHere's an analysis on different pieces relating to 'aa'\r\n```\r\n[root@apps current]# ifconfig eth1\r\neth1      Link encap:Ethernet  HWaddr 08:00:27:68:75:FA\r\n          inet addr:192.168.66.79  Bcast:192.168.66.255  Mask:255.255.255.0\r\n          inet6 addr: fe80::a00:27ff:fe68:75fa/64 Scope:Link\r\n          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1\r\n          RX packets:125468 errors:0 dropped:0 overruns:0 frame:0\r\n          TX packets:47958 errors:0 dropped:0 overruns:0 carrier:0\r\n          collisions:0 txqueuelen:1000\r\n          RX bytes:125161937 (119.3 MiB)  TX bytes:4089058 (3.8 MiB)\r\n```\r\n\r\nAccording to the aa.conf file, the port should be running locally on the apps server:\r\n```\r\n[root@apps current]# cat /etc/httpd/conf.d/aa.conf  |grep 9198\r\n    ProxyPass /aa/api http://localhost:9198/ retry=0\r\n    ProxyPassReverse /aa/api http://localhost:9198/\r\n```\r\n\r\n```\r\n[root@apps current]# /etc/init.d/aa status\r\naa (pid 30497) is running...\r\n```\r\n\r\n```\r\n[root@apps current]# telnet 127.0.0.1 9198\r\nTrying 127.0.0.1...\r\ntelnet: connect to address 127.0.0.1: Connection refused\r\n```\r\n\r\nThis is what I'm seeing in the  /var/log/aa/aa.log file:\r\n```\r\n2016-03-05 00:23:33,876 DEBUG [pool-4-thread-1] a.s.UrlResourceServiceRegistry:34 - Starting refreshing SP metadata.\r\n2016-03-05 00:23:33,877 DEBUG [pool-4-thread-1] a.s.UrlResourceServiceRegistry:51 - Fetching SP metadata entries from https://multidata.vm.openconext.org/service-providers.json\r\n2016-03-05 00:23:33,879 ERROR [pool-4-thread-1] a.s.UrlResourceServiceRegistry:68 - Error in refreshing metadata\r\njava.lang.RuntimeException: java.io.IOException: Server returned HTTP response code: 503 for URL: https://multidata.vm.openconext.org/service-providers.json\r\n        at aa.serviceregistry.ClassPathResourceServiceRegistry.readValue(ClassPathResourceServiceRegistry.java:80) ~[aa-server-1.0.1-20160302.080912-1.jar!/:1.0.1-SNAPSHOT]\r\n        at aa.serviceregistry.ClassPathResourceServiceRegistry.parseEntities(ClassPathResourceServiceRegistry.java:57) ~[aa-server-1.0.1-20160302.080912-1.jar!/:1.0.1-SNAPSHOT]\r\n        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_71]\r\n        at java.util.Collections$2.tryAdvance(Collections.java:4717) ~[na:1.8.0_71]\r\n        at java.util.Collections$2.forEachRemaining(Collections.java:4725) ~[na:1.8.0_71]\r\n        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_71]\r\n        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_71]\r\n        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_71]\r\n        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_71]\r\n        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_71]\r\n        at aa.serviceregistry.ClassPathResourceServiceRegistry.initializeMetadata(ClassPathResourceServiceRegistry.java:36) ~[aa-server-1.0.1-20160302.080912-1.jar!/:1.0.1-SNAPSHOT]\r\n        at aa.serviceregistry.UrlResourceServiceRegistry.initializeMetadata(UrlResourceServiceRegistry.java:58) ~[aa-server-1.0.1-20160302.080912-1.jar!/:1.0.1-SNAPSHOT]\r\n        at aa.serviceregistry.UrlResourceServiceRegistry.refreshMetataData(UrlResourceServiceRegistry.java:66) ~[aa-server-1.0.1-20160302.080912-1.jar!/:1.0.1-SNAPSHOT]\r\n        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_71]\r\n        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_71]\r\n        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_71]\r\n        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_71]\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_71]\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_71]\r\n        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_71]\r\nCaused by: java.io.IOException: Server returned HTTP response code: 503 for URL: https://multidata.vm.openconext.org/service-providers.json\r\n        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1840) ~[na:1.8.0_71]\r\n        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) ~[na:1.8.0_71]\r\n        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254) ~[na:1.8.0_71]\r\n        at aa.serviceregistry.BasicAuthenticationUrlResource.getInputStream(BasicAuthenticationUrlResource.java:32) ~[aa-server-1.0.1-20160302.080912-1.jar!/:1.0.1-SNAPSHOT]\r\n        at aa.serviceregistry.ClassPathResourceServiceRegistry.readValue(ClassPathResourceServiceRegistry.java:78) ~[aa-server-1.0.1-20160302.080912-1.jar!/:1.0.1-SNAPSHOT]\r\n        ... 19 common frames omitted\r\n```\r\n\r\nNo service listening on port 9198:\r\n```\r\n[root@apps current]# netstat -ln |grep 9198\r\n[root@apps current]# netstat -ln\r\nActive Internet connections (only servers)\r\nProto Recv-Q Send-Q Local Address               Foreign Address             State\r\ntcp        0      0 0.0.0.0:11211               0.0.0.0:*                   LISTEN\r\ntcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN\r\ntcp        0      0 127.0.0.1:9876              0.0.0.0:*                   LISTEN\r\ntcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN\r\ntcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN\r\ntcp        0      0 0.0.0.0:42939               0.0.0.0:*                   LISTEN\r\ntcp        0      0 0.0.0.0:389                 0.0.0.0:*                   LISTEN\r\ntcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN\r\ntcp        0      0 :::9291                     :::*                        LISTEN\r\ntcp        0      0 :::11211                    :::*                        LISTEN\r\ntcp        0      0 :::9196                     :::*                        LISTEN\r\ntcp        0      0 :::9197                     :::*                        LISTEN\r\ntcp        0      0 :::9199                     :::*                        LISTEN\r\ntcp        0      0 :::111                      :::*                        LISTEN\r\ntcp        0      0 :::22                       :::*                        LISTEN\r\ntcp        0      0 ::1:25                      :::*                        LISTEN\r\ntcp        0      0 :::33856                    :::*                        LISTEN\r\ntcp        0      0 :::389                      :::*                        LISTEN\r\ntcp        0      0 :::9190                     :::*                        LISTEN\r\ntcp        0      0 :::9191                     :::*                        LISTEN\r\ntcp        0      0 :::9290                     :::*                        LISTEN\r\nudp        0      0 0.0.0.0:111                 0.0.0.0:*\r\nudp        0      0 192.168.66.79:123           0.0.0.0:*\r\nudp        0      0 10.0.2.15:123               0.0.0.0:*\r\nudp        0      0 127.0.0.1:123               0.0.0.0:*\r\nudp        0      0 0.0.0.0:123                 0.0.0.0:*\r\nudp        0      0 0.0.0.0:60805               0.0.0.0:*\r\nudp        0      0 0.0.0.0:705                 0.0.0.0:*\r\nudp        0      0 0.0.0.0:68                  0.0.0.0:*\r\nudp        0      0 0.0.0.0:11211               0.0.0.0:*\r\nudp        0      0 0.0.0.0:724                 0.0.0.0:*\r\nudp        0      0 :::111                      :::*\r\nudp        0      0 fe80::a00:27ff:fe68:75fa:123 :::*\r\nudp        0      0 fe80::a00:27ff:fe5e:ac7:123 :::*\r\nudp        0      0 ::1:123                     :::*\r\nudp        0      0 :::123                      :::*\r\nudp        0      0 :::34307                    :::*\r\nudp        0      0 :::46738                    :::*\r\nudp        0      0 :::54814                    :::*\r\nudp        0      0 :::60959                    :::*\r\nudp        0      0 :::33956                    :::*\r\nudp        0      0 :::54316                    :::*\r\nudp        0      0 :::60590                    :::*\r\nudp        0      0 :::40880                    :::*\r\nudp        0      0 :::705                      :::*\r\nudp        0      0 :::48453                    :::*\r\nudp        0      0 :::11211                    :::*\r\n```"},"comment":{"url":"https://api.github.com/repos/OpenConext/OpenConext-deploy/issues/comments/193143422","html_url":"https://github.com/OpenConext/OpenConext-deploy/issues/48#issuecomment-193143422","issue_url":"https://api.github.com/repos/OpenConext/OpenConext-deploy/issues/48","id":193143422,"user":{"login":"quartje","id":14930136,"avatar_url":"https://avatars.githubusercontent.com/u/14930136?v=3","gravatar_id":"","url":"https://api.github.com/users/quartje","html_url":"https://github.com/quartje","followers_url":"https://api.github.com/users/quartje/followers","following_url":"https://api.github.com/users/quartje/following{/other_user}","gists_url":"https://api.github.com/users/quartje/gists{/gist_id}","starred_url":"https://api.github.com/users/quartje/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/quartje/subscriptions","organizations_url":"https://api.github.com/users/quartje/orgs","repos_url":"https://api.github.com/users/quartje/repos","events_url":"https://api.github.com/users/quartje/events{/privacy}","received_events_url":"https://api.github.com/users/quartje/received_events","type":"User","site_admin":false},"created_at":"2016-03-07T07:57:17Z","updated_at":"2016-03-07T07:57:17Z","body":"Hi Andrew,\r\n\r\nThis specific error should be fixed in the commit that has been done last Friday. \r\nThe aa component is dependent on the metadata-exporter component. That has been fixed, and aa should start up without complaining now!"}}