title: WSO2 Authentication failure on gateway node running WSO2 AM 2.1.0
body: <p>After working with WSO2 AM 1.10.0, we're now evaluating AM 2.1.0 for our WSO2 cluster. The cluster has two types of nodes:</p>
  
  <ul>
  <li>A manager node: a complete set of WSO2 AM products.</li>
  <li>A worker node: which runs only the gateway (-DworkerNode=true -Dprofile=gateway-worker)</li>
  </ul>
  
  <p>At the manager node, everything works ok. I am able to create a tenant (0000s7.com) and an API, and run it (I defined the GET verb to be unauthenticated, so no subscription is needed):</p>
  
  <pre><code>    $ curl http://localhost:8280/t/0000s7.com/ofer1/1.0.0
      { ...valid response... }
  </code></pre>
  
  <p>At the worker node:</p>
  
  <p>When I’m working with WSClient, I get the following error in the log:</p>
  
  <pre><code>TID: [-1234] [] [2018-02-06 07:26:48,893] ERROR {org.apache.axis2.transport.http.AxisServlet} -   
  {org.apache.axis2.transport.http.AxisServlet} java.lang.NullPointerException
          at org.apache.synapse.rest.RESTUtils.getFullRequestPath(RESTUtils.java:80)
          at org.apache.synapse.rest.API.canProcess(API.java:223)
          at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:97)
          at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:69)
          at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:304)
  </code></pre>
  
  <p>I also tried ThriftClient, with it I get a different error in the log:</p>
  
  <pre><code>   TID: [1] [] [2018-02-06 08:01:29,527] @0000s7.com [1] [AM] 
  WARN {org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClien    t} -  
  Login failed.. Authenticating again..{org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClient}
  </code></pre>
  
  <p>In both cases, API call ends with </p>
  
  <pre><code>&lt;ams:fault xmlns:ams="http://wso2.org/apimanager/security"&gt;
  &lt;ams:code&gt;900900&lt;/ams:code&gt;&lt;ams:message&gt;Unclassified Authentication Failure&lt;/ams:message&gt;
  &lt;ams:description&gt;Error while accessing backend services for API key validation&lt;/ams:description&gt;
  &lt;/ams:fault&gt;
  </code></pre>
  
  <p>Quick summary of changes. Currently I left the WSClient version to run:</p>
  
  <p><strong><em>Manager Node</em></strong> (internal IP 10.1.0.7)</p>
  
  <ul>
  <li><p>./repository/conf/api-manager.xml</p>
  
  <blockquote>
  <pre><code>        &lt;KeyValidatorClientType&gt;WSClient&lt;/KeyValidatorClientType&gt;
          &lt;ThriftClientConnectionTimeOut&gt;10000&lt;/ThriftClientConnectionTimeOut&gt;
          &lt;ThriftClientPort&gt;10397&lt;/ThriftClientPort&gt;
  
          &lt;EnableThriftServer&gt;false&lt;/EnableThriftServer&gt;
          &lt;ThriftServerHost&gt;10.1.0.7&lt;/ThriftServerHost&gt;
          &lt;ThriftServerPort&gt;10397&lt;/ThriftServerPort&gt;
  </code></pre>
  </blockquote></li>
  <li><p>./repository/conf/carbon.xml</p></li>
  </ul>
  
  <blockquote>
  <pre><code>      &lt;HostName&gt;10.1.0.7&lt;/HostName&gt;
        &lt;MgtHostName&gt;10.1.0.7&lt;/MgtHostName&gt;
  </code></pre>
  </blockquote>
  
  <ul>
  <li>./repository/conf/registry.xml</li>
  <li>./repository/conf/user-mgt.xml</li>
  </ul>
  
  <blockquote>
    <p>Minor changes to adopt to the MySQL DB</p>
  </blockquote>
  
  <ul>
  <li>./repository/conf/axis2/axis2.xml</li>
  </ul>
  
  <blockquote>
  <pre><code>            &lt;property name="subDomain" value="mgt"/&gt;
              &lt;property name="port.mapping.80" value="9763"/&gt;
              &lt;property name="port.mapping.443" value="9443"/&gt;
  </code></pre>
  </blockquote>
  
  <ul>
  <li>./repository/conf/datasources/master-datasources.xml</li>
  </ul>
  
  <blockquote>
  <pre><code>Changed to work with RDS using MySQL. 
  I'm using four databases:
  o regdb_21 (the “shared” registry”
  o apimgtdb_21
  o userdb_21
  o msgbrokerdb_21
  In addition, stats DB and local registry are maintained in H2.
  </code></pre>
  </blockquote>
  
  <p><strong><em>Worker Node</em></strong> (Internal IP 10.1.0.150)</p>
  
  <ul>
  <li>./bin/wso2server.sh</li>
  </ul>
  
  <blockquote>
  <pre><code>     -DworkerNode=true \
       -Dprofile=gateway-worker \
  </code></pre>
  </blockquote>
  
  <ul>
  <li>./repository/conf/api-manager.xml</li>
  </ul>
  
  <blockquote>
  <pre><code>        &lt;KeyValidatorClientType&gt;WSClient&lt;/KeyValidatorClientType&gt;
          &lt;ThriftClientPort&gt;10397&lt;/ThriftClientPort&gt;
          &lt;ThriftClientConnectionTimeOut&gt;10000&lt;/ThriftClientConnectionTimeOut&gt;
  
          &lt;ThriftServerHost&gt;10.1.0.7&lt;/ThriftServerHost&gt;
          &lt;ThriftServerPort&gt;10397&lt;/ThriftServerPort&gt;
          &lt;EnableThriftServer&gt;false&lt;/EnableThriftServer&gt;
          &lt;JMSConnectionDetails&gt;
              &lt;Enabled&gt;false&lt;/Enabled&gt;
              …
  
     Plus some changes of JMS IP to the IP address of the manager (10.1.0.7)
  </code></pre>
  </blockquote>
  
  <ul>
  <li>./repository/conf/carbon.xml</li>
  </ul>
  
  <blockquote>
  <pre><code>      &lt;HostName&gt;10.1.0.150&lt;/HostName&gt;
        &lt;MgtHostName&gt;10.1.0.150&lt;/MgtHostName&gt;
  </code></pre>
  </blockquote>
  
  <ul>
  <li>./repository/conf/registry.xml</li>
  <li>./repository/conf/user-mgt.xml</li>
  </ul>
  
  <blockquote>
  <pre><code>            Minor changes to adopt to the MySQL DB
  </code></pre>
  </blockquote>
  
  <ul>
  <li>./repository/conf/axis2/axis2.xml</li>
  </ul>
  
  <blockquote>
  <pre><code>         &lt;property name="port.mapping.80" value="9763"/&gt;
           &lt;property name="port.mapping.443" value="9443"/&gt;
  </code></pre>
  </blockquote>
  
  <ul>
  <li>./repository/conf/datasources/master-datasources.xml</li>
  </ul>
  
  <blockquote>
  <pre><code>        Same as in manager
  </code></pre>
  </blockquote>
  
  <ul>
  <li>./repository/conf/broker.xml</li>
  <li>./repository/conf/event-broker.xml</li>
  <li>./repository/conf/jndi.properties</li>
  <li>./repository/conf/messaging-event-broker.xml</li>
  </ul>
  
  <blockquote>
  <pre><code>  Changes in IP from localhost to 10.1.0.7
  </code></pre>
  </blockquote>
  
  <p>Complete logs</p>
  
  <ul>
  <li>Manager: <a href="http://thatsitradio.com/logs1/manager.html" rel="nofollow noreferrer">http://thatsitradio.com/logs1/manager.html</a></li>
  <li>Worker: <a href="http://thatsitradio.com/logs1/worker.html" rel="nofollow noreferrer">http://thatsitradio.com/logs1/worker.html</a></li>
  </ul>