Kubernetes 클러스터에 Prometheus / Grafana 설치.

monitoring namespace 생성.

kubectl create namespace monitoring

kubectl get namespace

NFS 스토리지 준비.

마스터 노드.

mkdir -p /mnt/prometheus /mnt/grafana
chmod 777 /mnt/prometheus /mnt/grafana

sudo apt-get install nfs-kernel-server -y

sudo bash -c 'cat >> /etc/exports << EOF
/mnt/prometheus 192.168.122.0/24(rw,sync,no_subtree_check,no_root_squash)
/mnt/grafana 192.168.122.0/24(rw,sync,no_subtree_check,no_root_squash)
EOF'

sudo systemctl restart nfs-server

sudo exportfs -v

워커 노드.

sudo apt-get update
sudo apt-get install -y nfs-common

NFS 마운트 테스트.

sudo mkdir -p /mnt/test

sudo mount -t nfs 192.168.122.10:/mnt/prometheus /mnt/test

mount | grep prometheus

touch /mnt/test/test-file

sudo umount /mnt/test

PV 생성.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 192.168.122.10
    path: /mnt/prometheus
  persistentVolumeReclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 192.168.122.10
    path: /mnt/grafana
  persistentVolumeReclaimPolicy: Retain

PVC 생성.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-pvc
  namespace: monitoring
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  volumeName: prometheus-pv
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
  namespace: monitoring
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  volumeName: grafana-pv

확인.

kubectl get pv

kubectl get pvc -n monitoring

Prometheus ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    scrape_configs:
      - job_name: 'kubernetes-cluster'
        kubernetes_sd_configs:
          - role: node
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

Prometheus Deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
      - name: prometheus
        image: prom/prometheus:latest
        args:
          - "--config.file=/etc/prometheus/prometheus.yml"
          - "--storage.tsdb.path=/prometheus"
        ports:
        - containerPort: 9090
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1000m"
        volumeMounts:
        - name: config
          mountPath: /etc/prometheus
        - name: storage
          mountPath: /prometheus
      volumes:
      - name: config
        configMap:
          name: prometheus-config
      - name: storage
        persistentVolumeClaim:
          claimName: prometheus-pvc

Prometheus Service.

apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  type: NodePort
  selector:
    app: prometheus
  ports:
  - port: 9090
    targetPort: 9090
    nodePort: 32664

Prometheus RBAC.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring

Grafana Deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - containerPort: 3000
        env:
        - name: GF_SECURITY_ADMIN_PASSWORD
          value: "admin"
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        volumeMounts:
        - name: storage
          mountPath: /var/lib/grafana
      volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: grafana-pvc

Grafana Service.

apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  type: NodePort
  selector:
    app: grafana
  ports:
  - port: 3000
    targetPort: 3000
    nodePort: 31211

Node Exporter DaemonSet.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9100'
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: node-exporter
        image: prom/node-exporter:latest
        ports:
        - containerPort: 9100
        args:
          - --path.procfs=/host/proc
          - --path.sysfs=/host/sys
          - --path.rootfs=/rootfs
          - --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
        - name: rootfs
          mountPath: /rootfs
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      - name: rootfs
        hostPath:
          path: /

Prometheus 설정 업데이트 후 재시작.

kubectl delete pod -n monitoring -l app=prometheus

상태 확인.

kubectl get pods -n monitoring

kubectl get svc -n monitoring

kubectl logs -n monitoring -l app=prometheus --tail=50

kubectl logs -n monitoring -l app=grafana --tail=50

Prometheus 확인.

curl -s http://192.168.122.10:32664/api/v1/query?query=up | head -20

Grafana 확인.

curl -s http://192.168.122.10:31211/api/health

Prometheus:
NodePort 32664

Grafana:
NodePort 31211

Grafana 로그인.

admin / admin

Prometheus datasource.

http://prometheus:9090

Node Exporter dashboard.

Grafana ID 1860

NFS 문제 확인.

kubectl describe pvc -n monitoring prometheus-pvc
kubectl describe pvc -n monitoring grafana-pvc

sudo systemctl status nfs-server

sudo exportfs -v

showmount -e 192.168.122.10