title: How do you get a Splunk forwarder to work with the main Splunk server?
body: <p>I have installed Splunk version 7.3.2 on one server.  I have installed Splunk forwarder version 7.3.2 on a second server.  Both servers are running Ubuntu 18.  </p>
  
  <p>On the Splunk forwarder I ran these commands:</p>
  
  <pre><code>sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/syslog -index main -sourcetype %app%
  
  sudo /opt/splunkforwarder/bin/splunk add forward-server x.x.x.x:9997
  </code></pre>
  
  <p>(where x.x.x.x is the IP address of the Splunk server)</p>
  
  <pre><code>sudo /opt/splunk/bin/splunk start
  </code></pre>
  
  <p>I tried rebooting both servers and restarting Splunk on both.</p>
  
  <p>From the Splunk dashboard I want to see some indication of the logs from the server with the forwarder.  But I don't see any. I log into the web UI, I go to Settings -> Monitoring Console ->  Indexing -> Indexes and Performance.  There is an "Instance" drop down menu.  But the only option I see is the Splunk server.  I do not see the forwarder.</p>
  
  <p>If I go to Data Inputs in the web UI, I cannot click "Next."</p>
  
  <p>How can I see some evidence that the Splunk web UI is receiving data from the Splunk forwarder?</p>
  
  <p>I check network connectivity, and nothing is blocking TCP/IP communication between the two.  I would expect to see some forwarded data in Splunk (on the main Splunk server), but I am not seeing that.  What should I do?</p>
  
  <p>Edit: Through the web UI for Splunk I configured a listening port for the forwarding on port 9997.  By running <code>splunk enable listen 9997</code> I get </p>
  
  <blockquote>
    <p>Failed to create. Configuration for port 9997 already exists.</p>
  </blockquote>
  
  <p>Update on 10/9/19</p>
  
  <p>Connectivity is configured over port 9997 between the two servers.  I have used nmap to test over this port to and from both servers using internal and external IP addresses.  Nothing is filtered over this port.  Therefore I am quite sure no firewall rule or security mechanism is to blame.  In fact, the back end splunk log for the main splunk server has registered some activity of the Splunk forwarder.</p>
  
  <p>I am trying to index /var/log/* on my Splunk forwarder server.  Here is an excerpt from /opt/splunkforwarder/var/log/splunk/splunkd.log: </p>
  
  <blockquote>
    <p>10-10-2019 00:21:18.059 +0000 INFO  WatchedFile - Will begin reading
    at offset=4835872 for file='/var/log/sampleoct.log'. ... 10-10-2019
    00:22:10.944 +0000 WARN  FileClassifierManager - The file
    '/var/log/.test123.swp' is invalid. Reason: binary. 10-10-2019
    00:22:10.944 +0000 INFO  TailReader - Ignoring file
    '/var/log/.test123.swp' due to: binary 10-10-2019 00:22:10.945 +0000
    WARN  FileClassifierManager - The file '/var/log/.test123.swp' is
    invalid. Reason: binary. ... 10-10-2019 00:30:50.948 +0000 INFO 
    TailReader - Ignoring file
    '/var/log/journal/94b0369aaba948b4b6a6b43288cee7e6/system.journal' due
    to: binary</p>
  </blockquote>
  
  <p>In /var/log/ there is a non-binary file of text that I created.  It is 106 KB large.  On the Splunk server, I see no evidence that a second instance (e.g., the Splunk forwarder) is working.</p>
  
  <p>On the Splunk server I see this in /opt/splunk/var/log/splunk/splunkd.log:</p>
  
  <blockquote>
    <p>10-10-2019 00:21:09.930 +0000 WARN  DateParserVerbose - Accepted time
    (Tue Oct  8 16:59:22 2019) is suspiciously far away from the previous
    event's time (Wed Oct  9 23:48:21 2019), but still accepted because it
    was extracted by the same pattern. Context:
    source=/var/log/test123123|host=ip-123-123-123-1|%%app%%|125
    10-10-2019 00:22:31.288 +0000 WARN  AggregatorMiningProcessor -
    Breaking event because limit of 256 has been exceeded -
    data_source="/var/log/test123", data_host="ip-123-123-123-1",
    data_sourcetype="%app%" 10-10-2019 00:22:31.288 +0000 WARN 
    AggregatorMiningProcessor - Changing breaking behavior for event
    stream because MAX_EVENTS (256) was exceeded without a single event
    break. Will set BREAK_ONLY_BEFORE_DATE to False, and unset any
    MUST_NOT_BREAK_BEFORE or MUST_NOT_BREAK_AFTER rules. Typically this
    will amount to treating this data as single-line only. -
    data_source="/var/log/test123", data_host="ip-123-123-123-1",
    data_sourcetype="%app%"</p>
  </blockquote>
  
  <p>Where in the web UI should I look for this log file?  It seems like it may be indexed, but I cannot find it.</p>