2026年5月

构建 Prometheus + Grafana 监控栈离线 Helm 包

适用场景：无法直接访问互联网的 Kubernetes 集群，通过 Helm 离线部署 kube-prometheus-stack（Prometheus Operator、Alertmanager、Grafana 等）以及 Loki 日志系统。

第一步：安装 Helm 客户端

在用于制作离线包的中转机器（Linux）上安装 Helm。

  1. 安装 Helm 3
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
  1. 验证安装
helm version
# 预期输出：version.BuildInfo{Version:"v3.x.x", ...}

第二步：配置 Helm 仓库

添加 Prometheus 社区仓库与 Grafana 官方仓库。

  1. 添加仓库
# Prometheus 社区仓库
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Grafana 官方仓库
helm repo add grafana https://grafana.github.io/helm-charts
  1. 更新索引
helm repo update

第三步：下载并补齐 Chart 依赖（核心）

建议在专门目录下操作。

  1. 创建工作目录
mkdir -p helm-offline && cd helm-offline
  1. 下载 kube-prometheus-stack 并更新依赖
# 拉取并解压主包
helm pull prometheus-community/kube-prometheus-stack --untar

# 进入目录下载 dependencies 中声明的子 Chart（如 node-exporter、kube-state-metrics 等）
cd kube-prometheus-stack
helm dependency update
cd ..
  1. 下载 loki 并更新依赖
helm pull grafana/loki --untar

cd loki
helm dependency update
cd ..
  1. 下载 promtail
helm pull grafana/promtail --untar

第四步：打包离线文件

完成后，helm-offline 目录下应包含以下三个目录：

  • kube-prometheus-stack/
  • loki/
  • promtail/

打包命令（示例）：

cd ..
zip -r helm-charts-all.zip \
  helm-offline/kube-prometheus-stack \
  helm-offline/loki \
  helm-offline/promtail
# values-low-resource.yaml
# 同级别目录创建 kube-prometheus-stack  loki  promtail
# 针对 2C4G 环境的极限优化配置（已修复重复定义 Bug + 镜像加速）

# 1. 开启 Alertmanager
alertmanager:
  enabled: true
  # --- 以下为内存优化配置 ---
  alertmanagerSpec:
    # 减少保留时间，降低内存占用
    retention: 4h
    # 限制 Alertmanager 自身的资源使用
    resources:
      requests:
        memory: 50Mi
        cpu: 10m
      limits:
        memory: 150Mi  # 限制在 150MB，防止吃光内存
        cpu: 100m
    nodeSelector:
      kubernetes.io/hostname: k3s-node-txy
  # 禁用持久化存储（低配环境建议使用内存存储，除非你需要长期保留告警历史）
  persistentVolume:
    enabled: false

# 2. Prometheus 资源限制与调度
prometheus:
  service:
    type: NodePort
  prometheusSpec:
    retention: 6h
    resources:
      requests:
        memory: 100Mi
        cpu: 50m
      limits:
        memory: 600Mi
        cpu: 500m
    nodeSelector:
      kubernetes.io/hostname: k3s-node-txy

# 3. Grafana 资源限制与调度
grafana:
  service:
    type: NodePort
  adminPassword: "admin"
  resources:
    requests:
      memory: 50Mi
      cpu: 10m
    limits:
      memory: 200Mi
      cpu: 100m
  nodeSelector:
    kubernetes.io/hostname: k3s-node-txy

# 4. 基础组件（合并了 enabled 和 image 配置）
kubeStateMetrics:
  enabled: true
  image:
    registry: registry.cn-hangzhou.aliyuncs.com
    repository: google_containers/kube-state-metrics
    tag: v2.18.0

prometheus-node-exporter:
  enabled: true

目标设备执行

# Helm安装
helm install prometheus ./kube-prometheus-stack     --namespace monitoring     --timeout 15m     --disable-openapi-validation     --no-hooks     -f values-low-resource.yaml

任务运行失败

执行安装时可能报错：

Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
* job prometheus-kube-prometheus-admission-create failed: BackoffLimitExceeded

  • ubernetes Job 的 backoffLimit 字段定义了 单个 Job 允许失败并重试的最大次数（默认值为 6）7。
  • 当 Job 中的 Pod 连续失败次数超过 backoffLimit 时，Job 会被标记为 Failed 状态，错误信息即为 BackoffLimitExceeded7。

这时候需要详细分析下原因

#  查看pod创建情况
kubectl get pods -n monitoring
NAME                                                     READY   STATUS    RESTARTS        AGE
prometheus-kube-prometheus-admission-create-8w4rc        0/1     Error     1 (35s ago)  

**# 查看Pod详细信息**
kubectl logs prometheus-kube-prometheus-admission-create-8w4rc
error: error from server (NotFound): pods "prometheus-kube-prometheus-admission-create-8w4rc" not found in namespace "default"
[root@k3s-master ~]# kubectl logs prometheus-kube-prometheus-admission-create-8w4rc  -n monitoring
W0729 15:43:19.423771       1 client_config.go:683] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
Error: failed to get secret: error getting secret: Get "https://10.43.0.1:443/api/v1/namespaces/monitoring/secrets/prometheus-kube-prometheus-admission": dial tcp 10.43.0.1:443: i/o timeout
Usage:
  kube-webhook-certgen create [flags]

Flags:
      --ca-name string       Name of ca file in the secret (default "ca")
      --cert-name string     Name of cert file in the secret (default "cert")
  -h, --help                 help for create
      --host string          Comma-separated hostnames and IPs to generate a certificate for
      --key-name string      Name of key file in the secret (default "key")
      --namespace string     Namespace of the secret where certificate information will be written
      --secret-name string   Name of the secret where certificate information will be written
      --secret-type string   Type of the secret where certificate information will be written (default "Opaque")

Global Flags:
      --kubeconfig string   Path to kubeconfig file: e.g. ~/.kube/kind-config-kind
      --log-format string   Log format: text|json (default "json")
      --log-level string    Log level: error|warn|info|debug (default "info")

{"time":"2026-07-29T15:43:49.450671764Z","level":"ERROR","source":{"function":"github.com/jkroepke/kube-webhook-certgen/cmd.Execute","file":"github.com/jkroepke/kube-webhook-certgen@v1.8.2/cmd/root.go","line":47},"msg":"failed to get secret: error getting secret: **Get \"****https://10.43.0.1:443/api/v1/namespaces/monitoring/secrets/prometheus-kube-prometheus-admission\":**** dial tcp 10.43.0.1:443: i/o timeout"}**

# 测试发现本Master节点无法和CoreDNS、API Server正常通信

以上报错是因为使用了自己创建的证书，导致CoreDNS、API Server异常

不要生成证书，否则容器会异常

恢复方式
# 1. 备份整个 tls 目录
sudo mv /var/lib/rancher/k3s/server/tls /var/lib/rancher/k3s/server/tls.bak

# 2. (可选但推荐) 删除动态生成的证书清单，确保所有证书都被重新创建
sudo rm -f /var/lib/rancher/k3s/server/cred/cert.json

# 3. 重新启动服务生成默认证书

镜像拉取失败（ImagePullBackOff）

现象：

prometheus-kube-state-metrics-...   0/1   ImagePullBackOff   ...

原因：

Failed to pull image "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0" ... i/o timeout

解决思路：通过国内镜像源提前拉取并打 Tag，使运行时仍按原镜像名找到本地镜像。

registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0 - 镜像下载 | registry.k8s.io

ctr images pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0

ctr images tag \
  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0 \
  registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0
  
ctr images pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/ghcr.io/jkroepke/kube-webhook-certgen:1.8.2

ctr images tag  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/ghcr.io/jkroepke/kube-webhook-certgen:1.8.2  ghcr.io/jkroepke/kube-webhook-certgen:1.8.2