<p>I am trying to find a DB (object storage) for my application. The application is really a wrapper over ISTIO Network Routing API. Basically simplifying the ISTIO configuratin for my network. Kubernetes (k8s) Custom Resource Definition (CRD) seems to fit my requirements. Also love the watch and REST API capability provided by CRD.</p>

<p>DB requirement</p>

<ul>
<li>Few 100 MBs data - worst case</li>
<li>Ability to watch objects</li>
<li>REST API support for object</li>
<li>Persistence</li>
<li>Around 2k writes/sec and similar/more reads/sec. Although I do have my application acting as a proxy for CRD where things can be cached.</li>
</ul>

<p>Why using CRD will be a good or a bad idea? Are there any performance implication using CRD. <a href="https://stackoverflow.com/questions/41063238/using-etcd-as-primary-store-database">This 2016 stackflow answer</a> suggest that etcd data is not in RAM. Whereas etcd link suggest that etcd can do <a href="https://github.com/etcd-io/etcd" rel="nofollow noreferrer">10k writes/sec</a>  (so even things are not in RAM and purely in disk, who cares). </p>

<p>I am seeing multiple application using k8s CRDs.</p>

<ul>
<li>Helm uses CRDs to store releases</li>
<li>Istio uses CRDs to store their networking routing API objects</li>
</ul>

<p>Considering that (<a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/" rel="nofollow noreferrer">CRD page</a>)</p>

<ul>
<li><p>A resource is an endpoint in the <strong>Kubernetes API</strong> that stores a collection of API objects of a certain kind. For example, the built-in pods resource contains a collection of Pod objects.</p></li>
<li><p>A custom resource is an extension of the <strong>Kubernetes API</strong> that is not necessarily available on every Kubernetes cluster. In other words, it represents a customization of a particular Kubernetes installation.</p></li>
</ul>

<p><a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions" rel="nofollow noreferrer">A <strong>CRD</strong></a> is for extending Kubernetes itself, it is not for application data.</p>

<p>The <a href="https://github.com/bitnami-labs/helm-crd/blob/7dea271427769f46cd710d5d9a57ae6af811ae23/examples/mariadb.yaml" rel="nofollow noreferrer">helm-crd/examples/mariadb.yaml</a> is about lightweight metadata which will enable Kubernetes to download the right release and through Helm install it.</p>

<p>It is not to store data for a random application which could exist <em>without</em> Kubernetes (as opposed to Helm releases, which make sense only in a Kubernetes deployment scenario)</p>

<p>Similarly, <a href="https://istio.io/docs/concepts/security/" rel="nofollow noreferrer">Istio CRD</a> makes sense only in a Kubernetes context:</p>

<blockquote>
  <p>Kubernetes currently implements the Istio configuration on Custom Resource Definitions (CRDs). These CRDs correspond to namespace-scope and cluster-scope CRDs and automatically inherit access protection via the Kubernetes RBAC. </p>
</blockquote>

<p>That approach (using etcd to store any application data) would not scale.</p>
