<p>According to <a href="https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/custom-metrics.md" rel="nofollow" title="Kubernetes Custom Metrics">Kubernetes Custom Metrics Proposal</a> containers can expose its app-level metrics in Prometheus format to be collected by Heapster. </p>

<p>Could anyone elaborate, if metrics are <em>pulled</em> by Heapster that means after the container terminates metrics for the last interval are lost? Can app <em>push</em> metrics to Heapster instead?</p>

<p>Or, is there a recommended approach to collect metrics from moderately short-lived containers running in Kubernetes?</p>

<p>Not to speak for the original author's intent, but I believe that proposal is primarily focused on custom metrics that you want to use for things like scheduling and autoscaling within the cluster, not for general purpose monitoring (for which as you mention, pushing metrics is sometimes critical).</p>

<p>There isn't a single recommended pattern for what to do with custom metrics in general. If your environment has a preferred monitoring stack or vendor, a common approach is to run a second container in each pod (a "sidecar" container) to push relevant metrics about the main container to your monitoring backend.</p>
