<p>I have a network with 3 ordering nodes running on a Kubernetes cluster. I am using NFS for persistence storage. The Kubernetes cluster is running on bare metal machines. I onboarded a new organization into the network, added that organization into the constortium. As soon as I execute d the command <code>peer channel create</code>, the orderer started throwing the below mentioned error:</p>
<pre><code>2021-01-31 11:09:04.412 UTC [orderer.consensus.etcdraft] createOrReadWAL -&gt; INFO 089 No WAL data found, creating new WAL at path '/var/hyperledger/production/orderer/etcdraft/wal/mvp1x-channel' channel=mvp1x-channel node=1
2021-01-31 11:09:04.549 UTC [orderer.consensus.etcdraft] createOrReadWAL -&gt; WARN 08a failed to create a temporary WAL directory channel=mvp1x-channel node=1 tmp-dir-path=/var/hyperledger/production/orderer/etcdraft/wal/mvp1x-channel.tmp dir-path=/var/hyperledger/production/orderer/etcdraft/wal/mvp1x-channel error=&quot;expected \&quot;/var/hyperledger/production/orderer/etcdraft/wal/mvp1x-channel.tmp\&quot; to be empty, got [\&quot;.nfs000000000982c7c700005443\&quot;]&quot;
2021-01-31 11:09:04.549 UTC [orderer.commmon.multichannel] newChainSupport -&gt; PANI 08b [channel: mvp1x-channel] Error creating consenter: failed to restore persisted raft data: failed to create or read WAL: failed to initialize WAL:expected &quot;/var/hyperledger/production/orderer/etcdraft/wal/mvp1x-channel.tmp&quot; to be empty, got [&quot;.nfs000000000982c7c700005443&quot;]
panic: [channel: mvp1x-channel] Error creating consenter: failed to restore persisted raft data: failed to create or read WAL: failed to initialize WAL:expected &quot;/var/hyperledger/production/orderer/etcdraft/wal/mvp1x-channel.tmp&quot; to be empty, got [&quot;.nfs000000000982c7c700005443&quot;]
</code></pre>
<p>After the automatic restart, it started throwing an error:</p>
<pre><code>2021-02-01 04:55:31.392 UTC [orderer.commmon.multichannel] newChain -&gt; PANI 1519 Error creating chain support: error creating consenter forchannel: mvp1x-channel: failed to restore persisted raft data: failed to create or read WAL: failed to open WAL: fileutil: file already locked
panic: Error creating chain support: error creating consenter for channel: mvp1x-channel: failed to restore persisted raft data: failed to create or read WAL: failed to open WAL: fileutil: file already locked

goroutine 74 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00014fe40, 0x0, 0x0, 0x0)
        /go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:230 +0x545
go.uber.org/zap.(*SugaredLogger).log(0xc0000101b0, 0xc000557004, 0x101b08c, 0x20, 0xc001ca7588, 0x1, 0x1, 0x0, 0x0, 0x0)
        /go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:234 +0x100
go.uber.org/zap.(*SugaredLogger).Panicf(...)
        /go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:159
github.com/hyperledger/fabric/common/flogging.(*FabricLogger).Panicf(...)
        /go/src/github.com/hyperledger/fabric/common/flogging/zap.go:74
github.com/hyperledger/fabric/orderer/common/multichannel.(*Registrar).newChain(0xc000146a00, 0xc000452b40)
        /go/src/github.com/hyperledger/fabric/orderer/common/multichannel/registrar.go:369 +0x251
github.com/hyperledger/fabric/orderer/common/multichannel.(*BlockWriter).WriteConfigBlock(0xc00056f540, 0xc00021cb40, 0xc000573ca0, 0x9, 0x9)
        /go/src/github.com/hyperledger/fabric/orderer/common/multichannel/blockwriter.go:118 +0x357
github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).writeConfigBlock(0xc001d7a000, 0xc00021cb40, 0x12)
        /go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain.go:1226 +0x28f
github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).writeBlock(0xc001d7a000, 0xc00021cb40, 0x12)
        /go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain.go:805 +0x1a9
github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).apply(0xc001d7a000, 0xc00059db10, 0x1, 0x1)
        /go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain.go:990 +0x24a
github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).run(0xc001d7a000)
        /go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain.go:713 +0x95c
created by github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).Start
        /go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain.go:360 +0x230
</code></pre>
<p>I believe the first error is thrown because the NFS client creates a file with pattern .nfsxxxxxxx. But I don't understand the second error and I am looking for a way to resolve it.</p>
<p>I am also looking for the other best storage mechanism alternative to the NFS server for a bare-metal cluster.</p>
<p>Any suggestion/feedback would be appreciated</p>
<p>Update: The other 2 orderers are running fine.</p>

Akshay Sood
<p>This bug is fixed in <a href="https://github.com/hyperledger/fabric/pull/2934" rel="nofollow noreferrer">https://github.com/hyperledger/fabric/pull/2934</a>, you can look at the issue <a href="https://github.com/hyperledger/fabric/issues/2931" rel="nofollow noreferrer">https://github.com/hyperledger/fabric/issues/2931</a></p>
