<p>I am currently working on a project that uses the Prometheus Server (Not the Prometheus Operator).
We're looking to introduce a way of modifying the PrometheusRules without having to redeploy it.</p>
<p>I'm completely new to containers and Kubernetes and a little over my head so I'm hoping someone could let me know if I'm wasting my time trying to make this work.</p>
<p>What I have thought of doing so far is:</p>
<ol>
<li>Store the PrometheusRules in a configmap.</li>
<li>Apply the configmap of rules to the Prometheus Server configuration.</li>
<li>Create a sidecar to the Prometheus Server that can modify this configMap.</li>
<li>The sidecar will have an API exposed so users will have CRUD functionality for the rules.</li>
<li>When successful modifying a rule, the sidecar will trigger the reload endpoint on the Prometheus Server that causes it to reload its configuration file without having to restart the container.</li>
</ol>
<p>Thanks</p>

Donncha
<p>Your initial use case seems valid, though I would say there are better ways of achieving this.</p>
<p>For points 1,2 I would suggest you use the <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus" rel="nofollow noreferrer">Prometheus Helm Chart</a> for ease of use, better config management and deployment. This would keep track of Prometheus configuration as one single unit rather than maintaining the rule files separately.</p>
<p>For point 3,4:- Making direct untracked changes to the live configuration does not seem safe or secure. Using the helm chart mentioned above I would suggest you make the changes <em>before</em> deploying to the cluster (Use VCS like Git to track changes)</p>
<p>Best case scenario:- Also setup CI/CD pipelines to deploy changes instantly.</p>
<p>Use the reload API as mentioned to reload the new released config.</p>
<p>Explore more about <a href="https://helm.sh/" rel="nofollow noreferrer">Helm</a></p>
