<p>I have deployed the hyperledger fabric blockchain on k8s with 3 peer nodes and 2 orderer nodes and 2 external chaincodes.</p>
<p><strong>Hyperledger Fabric v2.3.3 | Fabric-Peer v2.3</strong></p>
<p>For the peers I have <strong>assigned resources</strong> as:</p>
<pre><code>resources:
        limits:
          memory: &quot;800Mi&quot;
          cpu: &quot;350m&quot;
        requests:
          memory: &quot;750Mi&quot;
          cpu: &quot;300m&quot;
</code></pre>
<p>While doing some performance testing I have noticed the following:</p>
<ul>
<li>The <strong>memory usage keeps increasing</strong> with the number of concurrent transactions, which is expected given that the messages are basically protobufs and wrapped in an envelop.</li>
<li>I invoked about 500 concurrent transactions and soon the <strong>peer pods hit the memory limit</strong> and became unavailable which led to <strong>multiple failures</strong> - greylisting of peers, failure of endorsement policies etc.</li>
</ul>
<p>Now after sometimes, when there's no more transactions in the queue, then <em>peer should release the memory?</em> But, that's not happening, the peer pod is still using the maximum memory. <strong>Even after 1 hour the peer memory usage is maximum</strong></p>
<p>Also I have prometheus scraping all the metrics endpoints of peer and orderer nodes at an interval of 15s. And the client invokes one of the chaincode every 5 minutes (for some running total).</p>
<p>Are there any any environment settings which can be configured to get the optimal performance?</p>

Niraj Kumar
<p>What you can try to do for starters, is to disable gossip block dissemination (make <a href="https://github.com/hyperledger/fabric/blob/main/sampleconfig/core.yaml#L367" rel="nofollow noreferrer">this</a> key disabled) in all peers.</p>
<p>Gossip keeps an in-memory cache of recent blocks used for dissemination.</p>
<p>Also make sure the state transfer is <a href="https://github.com/hyperledger/fabric/blob/main/sampleconfig/core.yaml#L252" rel="nofollow noreferrer">disabled</a> and that all peers are configured to pull blocks from the orderer.</p>
