A sample containerized application in Kubernetes unable to be shown as targets in Prometheus for scraping metrics

My goal is to reproduce the observations in this blog post: https://medium.com/kubernetes-tutorials/monitoring-your-kubernetes-deployments-with-prometheus-5665eda54045

So far I am able to deploy the example rpc-app applicaiton in my cluster, the following shows the two pods for this application is running:

NAMESPACE     NAME                                                   READY   STATUS    RESTARTS   AGE     IP            NODE                           NOMINATED NODE   READINESS GATES
default       rpc-app-deployment-64f456b65-5m7j5                     1/1     Running   0          3h23m   10.244.0.15   my-server-ip.company.com   <none>           <none>
default       rpc-app-deployment-64f456b65-9mnfd                     1/1     Running   0          3h23m   10.244.0.14   my-server-ip.company.com   <none>           <none>

The application exposes metrics and is confirmed by:

root@xxxxx:/u01/app/k8s # curl 10.244.0.14:8081/metrics

    # HELP go_gc_duration_seconds A summary of the GC invocation durations.
    # TYPE go_gc_duration_seconds summary
    go_gc_duration_seconds{quantile="0"} 0
    go_gc_duration_seconds{quantile="0.25"} 0
        ...

    rpc_durations_seconds{service="uniform",quantile="0.5"} 0.0001021102787270781
    rpc_durations_seconds{service="uniform",quantile="0.9"} 0.00018233200374804932
    rpc_durations_seconds{service="uniform",quantile="0.99"} 0.00019828258205623097
    rpc_durations_seconds_sum{service="uniform"} 6.817882693745326
    rpc_durations_seconds_count{service="uniform"} 68279

My prometheus pod is running in the same cluster. However I am unable to see any rpc_* meterics in the prometheus.

monitoring    prometheus-deployment-599bbd9457-pslwf                 1/1     Running   0          30m     10.244.0.21   my-server-ip.company.com   <none>           <none>

In the promethus GUI

click Status -> Servcie Discovery, I got

Service Discovery rpc-metrics (0 / 3 active targets)

click Status -> Targets show nothing (0 targets)

click Status -> Configuration The content can be seen as: https://gist.github.com/denissun/14835468be3dbef7bc924032767b9d7f

I am really new to Prometheus/Kubernetes monitoring, appreciate your help to troubleshoot this issue.

from Recent Questions - Stack Overflow https://ift.tt/3sZxYWK
https://ift.tt/eA8V8J