title: command output not captured by shell script when invoked by snmp pass
body: <h1>The problem</h1>
  
  <p>SNMPD is correctly delegating SNMP polling requests to another program but the response from that program is not valid. A manual run of the program with the same arguments is responding correctly.</p>
  
  <h1>The detail</h1>
  
  <p>I've installed the correct LSI raid drivers on a server and want to configure SNMP. As per the instructions, I've added the following to <code>/etc/snmp/snmpd.conf</code> to redirect SNMP polling requests with a given OID prefix to a program:</p>
  
  <pre><code>pass .1.3.6.1.4.1.3582 /usr/sbin/lsi_mrdsnmpmain
  </code></pre>
  
  <p>It doesn't work correctly for SNMP polling requests:</p>
  
  <pre><code>snmpget -v1 -c public localhost .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.1
  </code></pre>
  
  <p>I get the following response:</p>
  
  <pre><code>Error in packet
  Reason: (noSuchName) There is no such variable name in this MIB.
  Failed object: SNMPv2-SMI::enterprises.3582.5.1.4.2.1.2.1.32.1
  </code></pre>
  
  <h1>What I've tried</h1>
  
  <p>SNMPD passes two arguments, <code>-g</code> and <code>&lt;oid&gt;</code> and expects a three line response <code>&lt;oid&gt;</code>, <code>&lt;data-type&gt;</code> and <code>&lt;data-value&gt;</code>.</p>
  
  <p>If I manually run the following:</p>
  
  <pre><code>/usr/sbin/lsi_mrdsnmpmain -g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0
  </code></pre>
  
  <p>I correctly get a correct three line response:</p>
  
  <pre><code>.1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0
  integer
  30
  </code></pre>
  
  <p>This means that the <code>pass</code> command is working correctly and the <code>/usr/sbin/lsi_mrdsnmpmain</code> program is working correctly <strong>in this example</strong></p>
  
  <p>I tried replacing <code>/usr/sbin/lsi_mrdsnmpmain</code> with a bash script. The bash script delegates the call and logs the supplied arguments and output from the delegated call:</p>
  
  <pre><code>#!/bin/bash
  echo "In: '$@" &gt; /var/log/snmp-pass-test
  RETURN=$(/usr/sbin/lsi_mrdsnmpmain $@)
  echo "$RETURN"
  echo "Out: '$RETURN'" &gt;&gt; /var/log/snmp-pass-test
  </code></pre>
  
  <p>And modified the <code>pass</code> command to redirect to the bash script. If I run the bash script manually <code>/usr/sbin/snmp-pass-test -g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0</code> I get the correct three line response as I did when I ran <code>/usr/sbin/lsi_mrdsnmpmain</code> manually and I get the following logged:</p>
  
  <pre><code>In: '-g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0
  Out: '.1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0
  integer
  30'
  </code></pre>
  
  <p>When I rerun the <code>snmpget</code> test, I get the same <code>Error in packet...</code> error and the bash script's logging shows that the captured delegated call output is empty:</p>
  
  <pre><code>In: '-g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0
  Out: ''
  </code></pre>
  
  <p>If I modify the bash script to only echo an empty line I also get the same <code>Error in packet...</code> message.</p>
  
  <p>I've also tried ensuring that the environment variables that are present when I manually call <code>/usr/sbin/lsi_mrdsnmpmain</code> are the same for the bash script but I get the same empty output.</p>
  
  <h1>Finally, my questions</h1>
  
  <ol>
  <li>Why would the bash script behave differently in these two scenarios?</li>
  <li>Is it likely that the problem that exists with the bash scripts is the same as originally noticed (manually running program has different output to SNMPD run program)?</li>
  </ol>
  
  <h2>Updates</h2>
  
  <h3>eewanco's suggestions</h3>
  
  <blockquote>
    <p>What user is running the program in each scenario?</p>
  </blockquote>
  
  <p>I added <code>echo "$(whoami)" &gt; /var/log/snmp-pass-test</code> to the bash script and <code>root</code> was added to the logs</p>
  
  <blockquote>
    <p>Maybe try executing it in cron</p>
  </blockquote>
  
  <p>Adding the following to root's crontab and the correct three line response was logged:</p>
  
  <pre><code>* * * * * /usr/sbin/lsi_mrdsnmpmain -g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.1 &gt;&gt; /var/log/snmp-test-cron 2&gt;&amp;1
  </code></pre>
  
  <h3>Grisha Levit's suggestion</h3>
  
  <blockquote>
    <p>Try logging the stderr</p>
  </blockquote>
  
  <p>There aren't any errors logged</p>
  
  <h3>Checking /var/log/messages</h3>
  
  <p>When I run it via SNMPD, I get <code>MegaRAID SNMP AGENT: Error in getting Shared Memory(lsi_mrdsnmpmain)</code> logged. When I run it directly, I don't. I've done a bit of googling and <a href="https://www.asus.com/support/faq/110704/" rel="nofollow noreferrer">I may need lm_sensors installed</a>; I'll try this.</p>
  
  <p>I installed lm_sensors &amp; compat-libstdc++-33.i686 (the latter because it said it was a pre-requisite from the instructions and I was missing it), uninstalled and reinstalled the LSI drivers and am experiencing the same issue.</p>
  
  <h3>SELinux</h3>
  
  <p>I accidently stumbled upon <a href="http://net-snmp.sourceforge.net/wiki/index.php/Tut:Extending_snmpd_using_shell_scripts" rel="nofollow noreferrer">a page about extending snmpd with scripts</a> and it says to check the script has the right SELinux context. I ran <code>grep AVC /var/log/audit/audit.log | grep snmp</code> before and after running a <code>snmpget</code> and the following entry is added as a direct result from running <code>snmpget</code>:</p>
  
  <pre><code>type=AVC msg=audit(1485967641.075:271): avc:  denied  { unix_read unix_write } for  pid=5552 comm="lsi_mrdsnmpmain" key=558265  scontext=system_u:system_r:snmpd_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=shm
  </code></pre>
  
  <p>I'm now <em>assuming</em> that SELinux is causing the call to fail; I'll dig further...see answer for solution.</p>
  
  <h3>strace (eewanco's suggestion)</h3>
  
  <blockquote>
    <p>Try using strace with and without snmp and see if you can catch a system call failure or some additional hints</p>
  </blockquote>
  
  <p>For completeness, I wanted to see if strace would have hinted that SELinux was denying. I had to remove the policy packages using <code>semodule -r &lt;policy-package-name&gt;</code> to reintroduce the problem then ran the following:</p>
  
  <pre><code>strace snmpget -v1 -c public localhost .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.1 &gt;&gt; strace.log 2&gt;&amp;1
  </code></pre>
  
  <p>The end of <code>strace.log</code> is as follows and unless I'm missing something, it doesn't seem to provide any hints:</p>
  
  <pre><code>...
  sendmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(161),     sin_addr=inet_addr("127.0.0.1")}, msg_iov(1)=    [{"0;\2\1\0\4\20public\240$\2\4I\264-m\2"..., 61}], msg_controllen=32,     {cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=, ...}, msg_flags=0},     MSG_DONTWAIT|MSG_NOSIGNAL) = 61
  select(4, [3], NULL, NULL, {0, 999997}) = 1 (in [3], left {0, 998475})
  brk(0xab9000)                           = 0xab9000
  recvmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(161),     sin_addr=inet_addr("127.0.0.1")}, msg_iov(1)=    [{"0;\2\1\0\4\20public\242$\2\4I\264-m\2"..., 65536}],     msg_controllen=0, msg_flags=0}, MSG_DONTWAIT) = 61
  write(2, "Error in packet\nReason: (noSuchN"..., 81Error in packet
  Reason: (noSuchName) There is no such variable name in this MIB.
  ) = 81
  write(2, "Failed object: ", 15Failed object: )         = 15
  write(2, "SNMPv2-SMI::enterprises.3582.5.1"..., 48SNMPv2-        SMI::enterprises.3582.5.1.4.2.1.2.1.32.1
  ) = 48
  write(2, "\n", 1
  )                       = 1
  brk(0xaa9000)                           = 0xaa9000
  close(3)                                = 0
  exit_group(2)                           = ?
  +++ exited with 2 +++
  </code></pre>