In additional to monitoring the health of your cluster, it is also important to monitor metrics about the applications themselves. This video discusses the basics of what it takes to get metric data from applications into Prometheus, as well as how to build custom Grafana dasboards on top of that data.
Here is the sample source code for the train schedule app that was used in the demo: https://github.com/linuxacademy/cicd-pipeline-train-schedule-monitoring
And here is a direct link to the Kubernetes template yml file that was used to deploy the app: https://github.com/linuxacademy/cicd-pipeline-train-schedule-monitoring/blob/master/train-schedule-kube.yml
sum(rate(http_request_duration_ms_count[2m])) by (service, route, method, code)