<p>My goal is to monitor services with Prometheus, so I was following a guide located at:</p>

<p><a href="https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md" rel="nofollow noreferrer">https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md</a></p>

<p>I am relatively new to all of this, so please forgive my naiveness. I tried looking into the error, but all the answers were convoluted. I have no idea where to start on the debug process (perhaps look into the YAMLs?)</p>

<p>I wanted to monitor a custom Service. So, I deployed a service.yaml of the following into a custom namespace (t): </p>

<pre><code>kind: Service
apiVersion: v1
metadata:
  namespace: t
  name: example-service-test
  labels:
    app: example-service-test
spec:
  selector:
    app: example-service-test
  type: NodePort
  ports:
  - name: http
    nodePort: 30901
    port: 8080
    protocol: TCP
    targetPort: http
---
apiVersion: v1
kind: Pod
metadata:
  name: example-service-test
  namespace: t
  labels:
    app: example-service-test
spec:
  containers:
  - name: example-service-test
    image: python:2.7
    imagePullPolicy: IfNotPresent
    command: ["/bin/bash"]
    args: ["-c", "echo \"&lt;p&gt;This is POD1 $(hostname)&lt;/p&gt;\" &gt; index.html; python -m SimpleHTTPServer 8080"]
    ports:
    - name: http
      containerPort: 8080
</code></pre>

<p>And deployed a service monitor into the namespace:</p>

<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-service-test
  labels:
    team: frontendtest1
  namespace: t
spec:
  selector:
    matchLabels:
      app: example-service-test
  endpoints:
  - port: http
</code></pre>

<p>So far, the service monitor is detecting the service, as shown: 
<a href="https://i.stack.imgur.com/CZsK2.png" rel="nofollow noreferrer">Prometheus Service Discovery</a>. 
However, there is an error with obtaining the metrics from the service: <a href="https://i.stack.imgur.com/BoNzI.png" rel="nofollow noreferrer">Prometheus Targets</a>. </p>

<p>From what I know, prometheus isn't able to access the /metrics on the sample service - in that case, do I need to expose the metrics? If so, could I get a step by step guide solution to how to expose metrics? If not, what route should I take? </p>

<p>I'm afraid you could miss the key thing from the tutorial you're following on CoreOS website, about how a metrics from an app are getting to Prometheus:</p>

<blockquote>
  <p>First, deploy three instances of a simple example application, <strong>which
  listens and exposes metrics on port 8080</strong></p>
</blockquote>

<p>Yes, your application (website) listens on port 8080, but does not expose any metrics on '/metrics' endpoint in the known to Prometheus format.</p>

<p>You can verify about what kind of metrics I'm talking about by hiting the endpoint from inside of Pod/Conatiner where it's hosted.</p>

<pre><code>kubectl exec -it $(kubectl get po -l app=example-app -o jsonpath='{.items[0].metadata.name}') -c example-app -- curl localhost:8080/metrics
</code></pre>

<p>You should see similar output to this one:</p>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-html lang-html prettyprint-override"><code># HELP codelab_api_http_requests_in_progress The current number of API HTTP requests in progress.
# TYPE codelab_api_http_requests_in_progress gauge
codelab_api_http_requests_in_progress 1
# HELP codelab_api_request_duration_seconds A histogram of the API HTTP request durations in seconds.
# TYPE codelab_api_request_duration_seconds histogram
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0001"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00015000000000000001"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00022500000000000002"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0003375"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00050625"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.000759375"} 0</code></pre>
</div>
</div>
</p>

<p>Please read more <a href="https://github.com/coreos/prometheus-operator/blob/master/Documentation/exposing-metrics.md" rel="nofollow noreferrer">here</a> on ways of exposing metrics. </p>
