Install Prometheus and Grafana on Kubernetes with Helm

Running Prometheus and Grafana on Kubernetes is the standard approach for monitoring clusters in production. The kube-prometheus-stack Helm chart (formerly known as prometheus-operator) bundles everything you need into a single deployment – Prometheus, Grafana, AlertManager, node-exporter, and kube-state-metrics. This guide walks through installing, configuring, and operating the full monitoring stack on a Kubernetes cluster.

I have been running this stack across multiple production clusters for years, and the kube-prometheus-stack chart has matured into the most reliable way to get comprehensive Kubernetes monitoring up and running quickly. It ships with sensible defaults, pre-built dashboards, and alerting rules that cover the majority of failure scenarios you will encounter.

Prerequisites

Before you begin, make sure you have the following in place:

• A running Kubernetes cluster (v1.25 or later) – any distribution works including EKS, GKE, AKS, k3s, or kubeadm clusters
• Helm 3 installed on your local machine
• kubectl configured to communicate with your cluster
• Cluster admin privileges (the chart creates CRDs, ClusterRoles, and other cluster-scoped resources)

Verify your cluster is accessible and Helm is installed by running these checks.

kubectl cluster-info
helm version

Both commands should return without errors. If kubectl cannot reach your cluster, fix your kubeconfig before proceeding. For guidance on setting up a Kubernetes cluster, check our guide on installing Kubernetes with kubeadm on Ubuntu.

What kube-prometheus-stack Includes

The kube-prometheus-stack Helm chart deploys a complete monitoring pipeline. Here is what gets installed when you deploy the chart:

• Prometheus Operator – manages Prometheus and AlertManager instances through Kubernetes CRDs (Custom Resource Definitions)
• Prometheus – the time-series database and metrics scraping engine
• Grafana – the visualization and dashboarding platform, pre-loaded with Kubernetes dashboards
• AlertManager – handles alert routing, grouping, and notification delivery
• node-exporter – exposes hardware and OS-level metrics from every node in the cluster
• kube-state-metrics – generates metrics about the state of Kubernetes objects (deployments, pods, nodes, etc.)
• Pre-configured alerting rules – dozens of production-grade alert rules for cluster health monitoring
• Pre-built Grafana dashboards – ready-to-use dashboards for cluster, node, pod, and workload metrics

The Prometheus Operator is the key component that makes everything work together. It introduces several CRDs – ServiceMonitor, PodMonitor, PrometheusRule, and AlertmanagerConfig – that let you manage monitoring configuration declaratively through Kubernetes manifests instead of editing Prometheus config files directly.

Add the Helm Repository and Install kube-prometheus-stack

Start by adding the prometheus-community Helm repository and updating your local chart index.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Create a dedicated namespace for the monitoring stack. Keeping monitoring resources in their own namespace makes access control and resource management straightforward.

kubectl create namespace monitoring

For a basic installation with default settings, you can install the chart directly.

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring

However, in production you will want to customize the installation. Create a values file that sets persistent storage, resource limits, and retention policies.

cat <<EOF > custom-values.yaml
prometheus:
  prometheusSpec:
    retention: 30d
    retentionSize: "45GB"
    resources:
      requests:
        memory: 2Gi
        cpu: 500m
      limits:
        memory: 4Gi
        cpu: "2"
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: standard
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi

grafana:
  adminPassword: "YourSecurePassword123"
  persistence:
    enabled: true
    size: 10Gi
    storageClassName: standard
  resources:
    requests:
      memory: 256Mi
      cpu: 100m
    limits:
      memory: 512Mi
      cpu: 500m

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: standard
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 5Gi
    resources:
      requests:
        memory: 128Mi
        cpu: 50m
      limits:
        memory: 256Mi
        cpu: 200m

nodeExporter:
  resources:
    requests:
      memory: 64Mi
      cpu: 50m
    limits:
      memory: 128Mi
      cpu: 100m

kubeStateMetrics:
  resources:
    requests:
      memory: 128Mi
      cpu: 50m
    limits:
      memory: 256Mi
      cpu: 100m
EOF

Now install the chart using your custom values file.

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  -f custom-values.yaml

Wait for all pods to reach the Running state. This typically takes two to three minutes depending on your cluster and image pull speeds.

kubectl get pods -n monitoring -w

You should see pods for Prometheus, Grafana, AlertManager, node-exporter (one per node), kube-state-metrics, and the Prometheus operator itself. All pods should show Ready status before you proceed.

Access the Grafana UI

Grafana is where you will spend most of your time viewing dashboards and building visualizations. There are three common ways to access the Grafana web interface.

Option 1 – Port Forward (Development and Testing)

The quickest way to access Grafana is through kubectl port-forward. This works well for development and quick checks but is not suitable for shared access.

kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80

Open your browser and navigate to http://localhost:3000. The default login credentials are:

• Username: admin
• Password: prom-operator (this is the default if you did not set a custom password in your values file)

If you set a custom admin password in your values file, use that instead.

Option 2 – NodePort Service

For environments where you want direct access through a node IP, you can patch the Grafana service to use NodePort.

kubectl patch svc kube-prometheus-stack-grafana -n monitoring \
  -p '{"spec": {"type": "NodePort", "ports": [{"port": 80, "nodePort": 31000}]}}'

Access Grafana at http://<any-node-ip>:31000. This approach works for on-premises clusters or lab environments where you have direct network access to nodes.

Option 3 – Ingress (Production)

For production clusters, expose Grafana through an Ingress resource. Add the following to your custom-values.yaml file and upgrade the release.

grafana:
  ingress:
    enabled: true
    ingressClassName: nginx
    hosts:
      - grafana.example.com
    tls:
      - secretName: grafana-tls
        hosts:
          - grafana.example.com

Make sure your Ingress controller is running and DNS points to the correct load balancer. For details on setting up an Ingress controller, see our guide on configuring Nginx Ingress Controller on Kubernetes.

Access the Prometheus UI

The Prometheus web interface is useful for running ad-hoc PromQL queries, checking target health, and verifying which alerting rules are active. Access it with a port-forward.

kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090

Open http://localhost:9090 in your browser. Navigate to Status > Targets to verify that all scrape targets are healthy. You should see targets for the API server, kubelet, node-exporter, kube-state-metrics, and the monitoring stack components themselves. Any target showing as DOWN needs investigation.

Try running a basic PromQL query to confirm metrics are flowing. Enter the following in the expression browser to see cluster-wide CPU usage.

sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (instance)

If you get results, your monitoring stack is working correctly and scraping node metrics.

Explore Default Dashboards in Grafana

One of the biggest advantages of the kube-prometheus-stack chart is the collection of pre-built Grafana dashboards it ships with. After logging into Grafana, navigate to Dashboards in the left sidebar. You will find a folder called “General” or “Default” containing several Kubernetes-specific dashboards.

The most useful dashboards for day-to-day operations include:

• Kubernetes / Compute Resources / Cluster – gives a high-level view of CPU and memory usage across the entire cluster, broken down by namespace
• Kubernetes / Compute Resources / Node (Pods) – shows resource consumption per node, helping identify hot spots and scheduling imbalances
• Kubernetes / Compute Resources / Pod – drills into individual pod metrics, useful for capacity planning and troubleshooting OOM kills
• Kubernetes / Compute Resources / Namespace (Workloads) – displays resource usage grouped by workload within a namespace
• Kubernetes / Networking / Cluster – covers network throughput, packet rates, and dropped packets across the cluster
• Node Exporter / Nodes – provides OS-level metrics like disk I/O, filesystem usage, network interface stats, and system load
• CoreDNS – monitors DNS query rates, latencies, and error rates for cluster DNS

These dashboards are provisioned as ConfigMaps and managed by the Grafana sidecar. Any changes you make in the Grafana UI will be lost on pod restart unless you save them to a persistent dashboard or a separate ConfigMap. I recommend using the built-in dashboards as a starting point and creating custom dashboards for your specific workloads.

Configure AlertManager for Notifications

The kube-prometheus-stack deploys AlertManager with a default configuration that does not route alerts anywhere useful. You need to configure receivers to send notifications to Slack, email, PagerDuty, or other services.

Add the AlertManager configuration to your custom-values.yaml file. Here is an example that routes critical alerts to Slack and sends everything else to email.

alertmanager:
  config:
    global:
      resolve_timeout: 5m
      smtp_smarthost: 'smtp.example.com:587'
      smtp_from: '[email protected]'
      smtp_auth_username: '[email protected]'
      smtp_auth_password: 'your-smtp-password'
      smtp_require_tls: true
    route:
      group_by: ['alertname', 'namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: 'email-notifications'
      routes:
        - receiver: 'slack-critical'
          match:
            severity: critical
          continue: true
        - receiver: 'email-notifications'
          match_re:
            severity: warning|critical
    receivers:
      - name: 'slack-critical'
        slack_configs:
          - api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
            channel: '#k8s-alerts'
            title: '{{ .GroupLabels.alertname }}'
            text: >-
              {{ range .Alerts }}
              *Alert:* {{ .Annotations.summary }}
              *Description:* {{ .Annotations.description }}
              *Severity:* {{ .Labels.severity }}
              *Namespace:* {{ .Labels.namespace }}
              {{ end }}
            send_resolved: true
      - name: 'email-notifications'
        email_configs:
          - to: '[email protected]'
            send_resolved: true

Apply the updated configuration by upgrading the Helm release.

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  -f custom-values.yaml

Verify AlertManager picked up the new config by port-forwarding to the AlertManager UI.

kubectl port-forward -n monitoring svc/kube-prometheus-stack-alertmanager 9093:9093

Open http://localhost:9093 and check the Status page to confirm your routing configuration is active. You can also trigger a test alert to verify end-to-end notification delivery.

Add Custom Prometheus Alerting Rules

The kube-prometheus-stack ships with a solid set of default alerting rules, but you will need to add custom rules for your own applications. The Prometheus Operator uses the PrometheusRule CRD for this purpose.

Create a PrometheusRule manifest for your custom alerts. The following example defines rules for high pod restart rates and persistent volume usage.

cat <<EOF | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: custom-app-rules
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: custom-application-alerts
      rules:
        - alert: HighPodRestartRate
          expr: |
            increase(kube_pod_container_status_restarts_total[1h]) > 5
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ \$labels.namespace }}/{{ \$labels.pod }} restarting frequently"
            description: "Pod {{ \$labels.pod }} in namespace {{ \$labels.namespace }} has restarted more than 5 times in the last hour."
        - alert: PersistentVolumeUsageCritical
          expr: |
            (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) * 100 > 90
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "PVC {{ \$labels.persistentvolumeclaim }} is over 90% full"
            description: "Persistent volume claim {{ \$labels.persistentvolumeclaim }} in namespace {{ \$labels.namespace }} is using {{ \$value | humanize }}% of its capacity."
        - alert: HighMemoryUsageByNamespace
          expr: |
            sum(container_memory_working_set_bytes{container!=""}) by (namespace) /
            sum(kube_resourcequota{type="hard", resource="limits.memory"}) by (namespace) * 100 > 85
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Namespace {{ \$labels.namespace }} is using over 85% of memory quota"
            description: "Memory usage in namespace {{ \$labels.namespace }} has exceeded 85% of the configured quota for more than 15 minutes."
EOF

The release: kube-prometheus-stack label is critical. The Prometheus Operator uses label selectors to discover PrometheusRule resources, and by default it looks for resources with this label matching the Helm release name. Without it, your rules will be ignored.

Confirm the rules were picked up by checking the Prometheus UI under Status > Rules, or query the API directly.

kubectl get prometheusrules -n monitoring

You should see your custom-app-rules resource listed alongside the default rules shipped with the chart.

Monitor External Services with ServiceMonitor

The ServiceMonitor CRD is how you tell Prometheus to scrape metrics from your own applications. Any service that exposes a /metrics endpoint in Prometheus format can be monitored this way.

Suppose you have an application running in the app-production namespace with a Service called my-api that exposes metrics on port 8080 at the /metrics path. Create a ServiceMonitor to have Prometheus scrape it.

cat <<EOF | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-api-monitor
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  namespaceSelector:
    matchNames:
      - app-production
  selector:
    matchLabels:
      app: my-api
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics
      scrapeTimeout: 10s
EOF

Again, the release: kube-prometheus-stack label must be present for the Prometheus Operator to pick up this ServiceMonitor. The namespaceSelector field tells Prometheus which namespace to look for the target service in, while selector matches the labels on the Kubernetes Service object.

If your application lives in the same namespace as the monitoring stack, you can omit the namespaceSelector field. For applications spread across many namespaces, you can also use namespaceSelector.any: true to match all namespaces, though this requires Prometheus to have RBAC permissions across the cluster.

You can also use a PodMonitor for pods that do not have an associated Service.

cat <<EOF | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: my-batch-job-monitor
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  namespaceSelector:
    matchNames:
      - batch-jobs
  selector:
    matchLabels:
      app: batch-processor
  podMetricsEndpoints:
    - port: metrics
      interval: 60s
      path: /metrics
EOF

After applying either resource, check the Prometheus Targets page to confirm the new target appears and shows an UP status.

Configure Persistent Storage for Prometheus and Grafana

Running Prometheus and Grafana without persistent storage means you lose all data when pods restart. In production, this is not acceptable. The custom-values.yaml file shown earlier already includes storage configuration, but here is a closer look at the key settings.

For Prometheus, storage is configured under the prometheusSpec.storageSpec section. The volume claim template creates a PVC that Prometheus uses for its TSDB (time-series database) data directory.

prometheus:
  prometheusSpec:
    retention: 30d
    retentionSize: "45GB"
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: standard
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi

Set retention and retentionSize to control how long data is kept. Prometheus will delete old data when either limit is reached, whichever comes first. Size your PVC to be larger than your retentionSize value to account for WAL (Write Ahead Log) and temporary compaction overhead.

For Grafana, persistent storage preserves dashboards, data sources, and user configurations across restarts.

grafana:
  persistence:
    enabled: true
    size: 10Gi
    storageClassName: standard
    accessModes:
      - ReadWriteOnce

Make sure the storageClassName matches a StorageClass available in your cluster. Check available StorageClasses with this command.

kubectl get storageclasses

For cloud providers, common values include gp3 for AWS EKS, standard-rwo for GKE, and managed-premium for AKS. For on-premises clusters using local storage or NFS, use the StorageClass name configured by your storage provisioner. If you need help setting up storage, see our guide on configuring persistent volumes in Kubernetes.

Upgrade the kube-prometheus-stack

Upgrading the monitoring stack is straightforward with Helm. First, update your local chart repository to pull the latest chart version.

helm repo update

Check what version you are currently running and what is available.

helm list -n monitoring
helm search repo prometheus-community/kube-prometheus-stack --versions | head -10

Before upgrading, review the chart changelog for any breaking changes. Major version bumps in the chart often include CRD updates that require attention. Since Helm does not manage CRD updates, you need to apply them manually before upgrading.

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/prometheus-community/helm-charts/main/charts/kube-prometheus-stack/charts/crds/crds/crd-prometheusrules.yaml

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/prometheus-community/helm-charts/main/charts/kube-prometheus-stack/charts/crds/crds/crd-servicemonitors.yaml

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/prometheus-community/helm-charts/main/charts/kube-prometheus-stack/charts/crds/crds/crd-podmonitors.yaml

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/prometheus-community/helm-charts/main/charts/kube-prometheus-stack/charts/crds/crds/crd-alertmanagerconfigs.yaml

Now run the upgrade with your existing values file.

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  -f custom-values.yaml

Monitor the rollout to make sure all pods come back healthy.

kubectl get pods -n monitoring -w

After the upgrade completes, verify the Prometheus and Grafana UIs are accessible and that your custom dashboards, rules, and ServiceMonitors are still intact. It is also good practice to check the Prometheus Targets page to confirm all scrape targets are UP.

Troubleshooting Common Issues

Running the monitoring stack in production, you will inevitably run into issues. Here are the most common problems and how to resolve them.

Targets Showing as DOWN

If scrape targets appear as DOWN in the Prometheus Targets page, start by checking whether Prometheus can reach the target endpoints. Common causes include:

• Network policies blocking traffic from the Prometheus pod to target pods
• Target pods not running or in CrashLoopBackOff state
• Incorrect port or path in the ServiceMonitor definition
• Missing RBAC permissions for Prometheus to discover endpoints in other namespaces

Check Prometheus operator logs for discovery errors.

kubectl logs -n monitoring -l app.kubernetes.io/name=prometheus-operator --tail=100

Also inspect the Prometheus pod logs for scrape errors.

kubectl logs -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -c prometheus --tail=100

Grafana Dashboards Showing No Data

When dashboards display “No data” panels, the issue is usually one of these:

• The Prometheus data source in Grafana is misconfigured – go to Configuration > Data Sources and verify the URL points to http://kube-prometheus-stack-prometheus.monitoring.svc:9090
• Time range is too narrow – expand the time range in the dashboard time picker
• Metric names changed between versions – check that the PromQL queries in the dashboard panels reference valid metric names
• The specific component (node-exporter, kube-state-metrics) is not running or not being scraped

Test connectivity from within the cluster by exec-ing into the Grafana pod.

kubectl exec -n monitoring -it deploy/kube-prometheus-stack-grafana -- \
  wget -qO- http://kube-prometheus-stack-prometheus.monitoring.svc:9090/api/v1/query?query=up | head -c 500

If this returns metric data, the connection is fine and the issue is in the dashboard queries. If it fails, check network policies and service DNS resolution.

AlertManager Not Sending Notifications

If alerts fire in Prometheus but notifications never arrive, work through these checks:

First, verify that alerts are actually reaching AlertManager. Port-forward to the AlertManager UI and check whether alerts appear on the main page.

kubectl port-forward -n monitoring svc/kube-prometheus-stack-alertmanager 9093:9093

If alerts are visible in AlertManager but notifications are not being sent:

• Check AlertManager logs for delivery errors – SMTP auth failures, Slack webhook rejections, and similar issues will appear here
• Verify the route configuration matches the alert labels – a mismatch between route matchers and alert labels will cause alerts to fall through to the default receiver
• Check that the receiver configuration is correct – Slack webhook URLs expire, SMTP credentials rotate, and PagerDuty keys change
• Look for inhibition rules that might be silencing the alert

View AlertManager logs with this command.

kubectl logs -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 --tail=200

Custom ServiceMonitor or PrometheusRule Not Being Picked Up

The most common reason for Prometheus ignoring your custom CRDs is missing or incorrect labels. The Prometheus Operator uses label selectors to discover these resources. Verify the selector configuration.

kubectl get prometheus -n monitoring -o yaml | grep -A 10 serviceMonitorSelector

The output shows which labels Prometheus looks for when discovering ServiceMonitors. Your ServiceMonitor must carry matching labels. The same applies to PrometheusRule resources – check ruleSelector in the Prometheus spec.

Prometheus Running Out of Memory

Prometheus memory usage scales with the number of active time series. If Prometheus pods are getting OOMKilled, you have a few options:

• Increase memory limits in your values file
• Reduce scrape frequency by increasing the interval in ServiceMonitors
• Drop high-cardinality metrics using metric_relabel_configs
• Reduce the retention period to keep fewer data points

Check the current number of active series with this PromQL query in the Prometheus UI.

prometheus_tsdb_head_series

As a rough guideline, Prometheus needs approximately 1-2 KB of memory per active time series. If you have 1 million active series, expect Prometheus to use around 1-2 GB of memory for the in-memory index alone.

Summary

The kube-prometheus-stack Helm chart gives you a production-ready monitoring setup with minimal effort. It handles the complex wiring between Prometheus, Grafana, AlertManager, and the various exporters, letting you focus on defining what to monitor and how to respond to alerts. The Prometheus Operator CRDs – ServiceMonitor, PodMonitor, PrometheusRule, and AlertmanagerConfig – make it possible to manage monitoring configuration alongside your application manifests, which fits naturally into GitOps workflows.

For ongoing operations, keep your custom-values.yaml file in version control, test upgrades in a staging environment before applying them to production, and review the pre-built alerting rules periodically to make sure they still make sense for your infrastructure. A well-maintained monitoring stack is the foundation for running reliable services on Kubernetes.