title: why do nvprof and nvidia-smi report different results on power?
body: <p>I used nvprof and nvidia-smi to monitor the GPU power dissipation respectively, but observed different results, summarized in the table below.</p>
  
  <pre><code>----------------------------------------------------------------
  gpu     |             busy           |             idle         
  model   |  nvprof[Watt]  smi[Watt]   |  nvprof[Watt]  smi[Watt] 
  ----------------------------------------------------------------
  M2090   |   ~151           ~151      |     ~100          ~75
  K20     |   ~105           ~102      |     ~63           ~43
  ----------------------------------------------------------------
  </code></pre>
  
  <p>note 0: "busy" means my code is running on the monitored GPU</p>
  
  <p>note 1: nvprof reports the power for all the devices. So my way to get the "idle" power using nvprof for a specific GPU is simply to have the code running on another GPU.</p>
  
  <p>note 2: nvidia-smi reports a couple of different quantities about power, but I was focusing on "power draw"</p>
  
  <p>note 3: cuda version: 5.5</p>
  
  <p>So my question is: why is the power reported by nvidia-smi generally smaller than nvprof, and why does this discrepancy become larger when the idle power is monitored? and ultimately, which utility should I trust more?</p>
  
  <p>Also, just to make sure, does the power that the two utilies measure refer to the input electric power (P=I*U) rather than the output thermal power, right?</p>
  
  <p>Thanks a lot for any advice! </p>
  
  <hr>
  
  <p><strong>Update</strong>
  @njuffa and @talonmies 's speculation makes very good sense. So I explored smi a little bit more for power analysis. The results, however, do not make sense to me.</p>
  
  <p><img src="https://i.stack.imgur.com/9Azja.png" alt="enter image description here"></p>
  
  <p>additional notes:</p>
  
  <ol>
  <li><p>The discontinuity of the red data is because I directly used the
  timestamp reported by smi, which has low resolution (sec). Besides,
  for illustration purpose p0 is assigned an numerical value of 20 and
  p1 of 10. So for most of the time, the GPU is put into its full
  performance state (this is <strong>odd</strong>), except for the "busy" case, where the GPU somehow
  drops to p1 during 15~18s (<strong>odd</strong>).</p></li>
  <li><p>It is not until ~21.3s that cudaSetDevice() is invoked for the very
  first time. So the power rise and p-state change that occurs at ~18s
  is rather <strong>odd</strong>.</p></li>
  <li><p>"busy power" is measured when my GPU code is set to the background,
  and smi put into an infinite loop to query the power and p-state
  repeatedly until the background process terminates. "idle power" is
  measured simply by launching smi 50 times. Apparently in the latter
  case, smi exhibits larger overhead, which is again, <strong>odd</strong>.</p></li>
  </ol>