构建 Prometheus + Grafana 监控栈离线 Helm 包
适用场景：无法直接访问互联网的 Kubernetes 集群，通过 Helm 离线部署 kube-prometheus-stack（Prometheus Operator、Alertmanager、Grafana 等）以及 Loki 日志系统。
第一步：安装 Helm 客户端
在用于制作离线包的中转机器（Linux）上安装 Helm。
- 安装 Helm 3
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
- 验证安装
helm version
# 预期输出：version.BuildInfo{Version:"v3.x.x", ...}
第二步：配置 Helm 仓库
添加 Prometheus 社区仓库与 Grafana 官方仓库。
- 添加仓库
# Prometheus 社区仓库
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
# Grafana 官方仓库
helm repo add grafana https://grafana.github.io/helm-charts
- 更新索引
helm repo update
第三步：下载并补齐 Chart 依赖（核心）
建议在专门目录下操作。
- 创建工作目录
mkdir -p helm-offline && cd helm-offline
- 下载 kube-prometheus-stack 并更新依赖
# 拉取并解压主包
helm pull prometheus-community/kube-prometheus-stack --untar
# 进入目录下载 dependencies 中声明的子 Chart（如 node-exporter、kube-state-metrics 等）
cd kube-prometheus-stack
helm dependency update
cd ..
- 下载 loki 并更新依赖
helm pull grafana/loki --untar
cd loki
helm dependency update
cd ..
- 下载 promtail
helm pull grafana/promtail --untar
第四步：打包离线文件
完成后，helm-offline 目录下应包含以下三个目录：
kube-prometheus-stack/loki/promtail/
打包命令（示例）：
cd ..
zip -r helm-charts-all.zip \
helm-offline/kube-prometheus-stack \
helm-offline/loki \
helm-offline/promtail
# values-low-resource.yaml
# 同级别目录创建 kube-prometheus-stack loki promtail
# 针对 2C4G 环境的极限优化配置（已修复重复定义 Bug + 镜像加速）
# 1. 开启 Alertmanager
alertmanager:
enabled: true
# --- 以下为内存优化配置 ---
alertmanagerSpec:
# 减少保留时间，降低内存占用
retention: 4h
# 限制 Alertmanager 自身的资源使用
resources:
requests:
memory: 50Mi
cpu: 10m
limits:
memory: 150Mi # 限制在 150MB，防止吃光内存
cpu: 100m
nodeSelector:
kubernetes.io/hostname: k3s-node-txy
# 禁用持久化存储（低配环境建议使用内存存储，除非你需要长期保留告警历史）
persistentVolume:
enabled: false
# 2. Prometheus 资源限制与调度
prometheus:
service:
type: NodePort
prometheusSpec:
retention: 6h
resources:
requests:
memory: 100Mi
cpu: 50m
limits:
memory: 600Mi
cpu: 500m
nodeSelector:
kubernetes.io/hostname: k3s-node-txy
# 3. Grafana 资源限制与调度
grafana:
service:
type: NodePort
adminPassword: "admin"
resources:
requests:
memory: 50Mi
cpu: 10m
limits:
memory: 200Mi
cpu: 100m
nodeSelector:
kubernetes.io/hostname: k3s-node-txy
# 4. 基础组件（合并了 enabled 和 image 配置）
kubeStateMetrics:
enabled: true
image:
registry: registry.cn-hangzhou.aliyuncs.com
repository: google_containers/kube-state-metrics
tag: v2.18.0
prometheus-node-exporter:
enabled: true
目标设备执行
# Helm安装
helm install prometheus ./kube-prometheus-stack --namespace monitoring --timeout 15m --disable-openapi-validation --no-hooks -f values-low-resource.yaml
任务运行失败
执行安装时可能报错：
Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
* job prometheus-kube-prometheus-admission-create failed: BackoffLimitExceeded
- ubernetes Job 的
backoffLimit 字段定义了 单个 Job 允许失败并重试的最大次数（默认值为 6）7。 - 当 Job 中的 Pod 连续失败次数超过
backoffLimit 时，Job 会被标记为 Failed 状态，错误信息即为 BackoffLimitExceeded7。
这时候需要详细分析下原因
# 查看pod创建情况
kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
prometheus-kube-prometheus-admission-create-8w4rc 0/1 Error 1 (35s ago)
**# 查看Pod详细信息**
kubectl logs prometheus-kube-prometheus-admission-create-8w4rc
error: error from server (NotFound): pods "prometheus-kube-prometheus-admission-create-8w4rc" not found in namespace "default"
[root@k3s-master ~]# kubectl logs prometheus-kube-prometheus-admission-create-8w4rc -n monitoring
W0729 15:43:19.423771 1 client_config.go:683] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
Error: failed to get secret: error getting secret: Get "https://10.43.0.1:443/api/v1/namespaces/monitoring/secrets/prometheus-kube-prometheus-admission": dial tcp 10.43.0.1:443: i/o timeout
Usage:
kube-webhook-certgen create [flags]
Flags:
--ca-name string Name of ca file in the secret (default "ca")
--cert-name string Name of cert file in the secret (default "cert")
-h, --help help for create
--host string Comma-separated hostnames and IPs to generate a certificate for
--key-name string Name of key file in the secret (default "key")
--namespace string Namespace of the secret where certificate information will be written
--secret-name string Name of the secret where certificate information will be written
--secret-type string Type of the secret where certificate information will be written (default "Opaque")
Global Flags:
--kubeconfig string Path to kubeconfig file: e.g. ~/.kube/kind-config-kind
--log-format string Log format: text|json (default "json")
--log-level string Log level: error|warn|info|debug (default "info")
{"time":"2026-07-29T15:43:49.450671764Z","level":"ERROR","source":{"function":"github.com/jkroepke/kube-webhook-certgen/cmd.Execute","file":"github.com/jkroepke/kube-webhook-certgen@v1.8.2/cmd/root.go","line":47},"msg":"failed to get secret: error getting secret: **Get \"****https://10.43.0.1:443/api/v1/namespaces/monitoring/secrets/prometheus-kube-prometheus-admission\":**** dial tcp 10.43.0.1:443: i/o timeout"}**
# 测试发现本Master节点无法和CoreDNS、API Server正常通信
以上报错是因为使用了自己创建的证书，导致CoreDNS、API Server异常
不要生成证书，否则容器会异常
恢复方式
# 1. 备份整个 tls 目录
sudo mv /var/lib/rancher/k3s/server/tls /var/lib/rancher/k3s/server/tls.bak
# 2. (可选但推荐) 删除动态生成的证书清单，确保所有证书都被重新创建
sudo rm -f /var/lib/rancher/k3s/server/cred/cert.json
# 3. 重新启动服务生成默认证书
镜像拉取失败（ImagePullBackOff）
现象：
prometheus-kube-state-metrics-... 0/1 ImagePullBackOff ...
原因：
Failed to pull image "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0" ... i/o timeout
解决思路：通过国内镜像源提前拉取并打 Tag，使运行时仍按原镜像名找到本地镜像。
registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0 - 镜像下载 | registry.k8s.io
ctr images pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0
ctr images tag \
swr.cn-north-4.myhuaweicloud.com/ddn-k8s/registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0 \
registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0
ctr images pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/ghcr.io/jkroepke/kube-webhook-certgen:1.8.2
ctr images tag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/ghcr.io/jkroepke/kube-webhook-certgen:1.8.2 ghcr.io/jkroepke/kube-webhook-certgen:1.8.2