<p>I'm trying to get rid of a bunch of orphaned volumes in heketi. When I try, I get "Error" and then information about the volume I just tried to delete, serialized as JSON. There's nothing else. I've tried to dig into the logs but they don't reveal anything.</p>

<p>This is the command I used to try and delete the volume:</p>

<pre><code>heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret ${KEY}  volume delete 22f1a960651f0f16ada20a15d68c7dd6
Error: {"size":30,"name":"vol_22f1a960651f0f16ada20a15d68c7dd6","durability":{"type":"none","replicate":{},"disperse":{}},"gid":2008,"glustervolumeoptions":["","cluster.post-op-delay-secs 0"," performance.client-io-threads off"," performance.open-behind off"," performance.readdir-ahead off"," performance.read-ahead off"," performance.stat-prefetch off"," performance.write-behind off"," performance.io-cache off"," cluster.consistent-metadata on"," performance.quick-read off"," performance.strict-o-direct on"," storage.health-check-interval 0",""],"snapshot":{"enable":true,"factor":1},"id":"22f1a960651f0f16ada20a15d68c7dd6","cluster":"e924a50aa93d9eae1132c60eb1f36310","mount":{"glusterfs":{"hosts":["&lt;SECRET&gt;"],"device":"&lt;SECRET&gt;:vol_22f1a960651f0f16ada20a15d68c7dd6","options":{"backup-volfile-servers":""}}},"blockinfo":{},"bricks":[{"id":"0f4c6d7f605e9368bfe3dc7cc117b69a","path":"/var/lib/heketi/mounts/vg_970f0faf60f8dfc6f6a0d6bd25bdea7c/brick_0f4c6d7f605e9368bfe3dc7cc117b69a/brick","device":"970f0faf60f8dfc6f6a0d6bd25bdea7c","node":"107894a855c9d2c34509b18272e6c298","volume":"22f1a960651f0f16ada20a15d68c7dd6","size":31457280}]}
</code></pre>

<p>Notice that the second line only contains Error, then the info about the volume serialized as json.</p>

<p>The volume doesn't exist in gluster. I used the below commands to verify the volume was no longer there:</p>

<pre><code>kubectl -n default  exec -t -i glusterfs-rgz9g bash
gluster volume info 
&lt;shows volume i did not delete&gt;
</code></pre>

<p>Kubernetes does not show a PersistentVolumeClaim or PersistentVolume:</p>

<pre><code>kubectl get pvc -A
No resources found.
kubectl get pv -A
No resources found.
</code></pre>

<p>I tried looking at the heketi logs, but it only reports a GET for the volume</p>

<pre><code>kubectl -n default logs   heketi-56f678775c-nrbwd
[negroni] 2019-11-25T21:29:19Z | 200 |   1.407715ms | &lt;SECRET&gt;:8080 | GET /volumes/22f1a960651f0f16ada20a15d68c7dd6
[negroni] 2019-11-25T21:29:19Z | 200 |   1.111984ms | &lt;SECRET&gt;:8080 | GET /volumes/22f1a960651f0f16ada20a15d68c7dd6
[negroni] 2019-11-25T21:29:19Z | 200 |   1.540357ms | &lt;SECRET&gt;:8080 | GET /volumes/22f1a960651f0f16ada20a15d68c7dd6 
</code></pre>

<p>I've tried setting more verbose log level but the setting doesn't stick:</p>

<pre><code>heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret ${KEY}  loglevel set debug
Server log level updated
heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret ${KEY}  loglevel get
info
</code></pre>

<p>My CLI uses</p>

<pre><code>heketi-cli -v
heketi-cli v9.0.0
</code></pre>

<p>And the Heketi server is running:</p>

<pre><code>kubectl -n default  exec -t -i  heketi-56f678775c-nrbwd bash
heketi -v
Heketi v9.0.0-124-gc2e2a4ab
</code></pre>

<p>Based on the logs, I believe heketi-cli has an issue and then never actually sends the POST or DELETE request to the heketi server.</p>

<p>How do I proceed to debug this? At this point my only work around is to recreate my cluster but I'd like to avoid that especially if something like this comes back. </p>

<p>Looks like there's a bug in <code>heketi-cli</code> because if I manually craft the request using ruby and curl, I'm able to delete the volume:</p>

<pre><code>TOKEN=$(ruby makeToken.rb DELETE /volumes/22f1a960651f0f16ada20a15d68c7dd6)
curl -X DELETE -H "Authorization: Bearer $TOKEN" http://10.233.21.178:8080/volumes/22f1a960651f0f16ada20a15d68c7dd6
</code></pre>

<p>Please see <a href="https://github.com/heketi/heketi/blob/master/docs/api/api.md#authentication-model" rel="nofollow noreferrer">https://github.com/heketi/heketi/blob/master/docs/api/api.md#authentication-model</a> for how to generate the jwt token.</p>

<p>I manually created the request, expected getting a better error message that the command line tool swallowed. Turns out the cli was actually busted.</p>

<p>ruby code for making the jwt token (makeToken.rb). You need to fill in pass and server.</p>

<pre><code>#!/usr/bin/env ruby

require 'jwt'
require 'digest'

user = "admin"
pass = "&lt;SECRET&gt;"
server = "http://localhost:8080/"

method = "#{ARGV[0]}"
uri = "#{ARGV[1]}"

payload = {}

headers = {
  iss: 'admin',
  iat: Time.now.to_i,
  exp: Time.now.to_i + 600,
  qsh: Digest::SHA256.hexdigest("#{method}&amp;#{uri}")
}

token = JWT.encode headers, pass, 'HS256'
print("#{token}")
</code></pre>
