磁盘 fdisk

fdisk

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
❯ sudo fdisk  /dev/nvme0n1 -l
Disk /dev/nvme0n1: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: KINGSTON SNV2S2000G
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 567141B0-39E4-467E-B67D-61B05D225406

Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 117186559 117184512 55.9G Linux filesystem
/dev/nvme0n1p2 117186560 234373119 117186560 55.9G Linux filesystem
/dev/nvme0n1p3 234373120 644530175 410157056 195.6G Linux filesystem
/dev/nvme0n1p4 644530176 1386717183 742187008 353.9G Linux filesystem
/dev/nvme0n1p5 1386717184 1402341375 15624192 7.5G Linux swap
/dev/nvme0n1p6 1402341376 1403473919 1132544 553M EFI System
/dev/nvme0n1p7 1403473920 1571248127 167774208 80G Microsoft basic data

创建新分区
分配未使用的分区
Command (m for help): F
Unpartitioned space /dev/nvme0n1: 1.09 TiB, 1195919875584 bytes, 2335781007 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes

Start End Sectors Size 1571248128 3907029134 2335781007 1.1T
Command (m for help): n Partition number (8-128, default 8): First sector (1571248128-3907029134, default 1571248128): Last sector, +/-sectors or +/-size{K,M,G,T,P} (1571248128-3907029134, default 3907029134): +80G
Created a new partition 8 of type 'Linux filesystem' and of size 80 GiB.

uuid

promethus

Prometheus

Prometheus内部主要分为三大块:

  • Retrieval是负责定时去暴露的目标页面上去抓取采样指标数据
  • Storage是负责将采样数据写磁盘
  • PromQL是Prometheus提供的查询语言模块

数据模型

  • Metric name(指标名称):该名字应该具有语义,一般用于表示 metric 的功能,例如:http_requests_total, 表示 http 请求的总数。其中,metric 名字由 ASCII 字符,数字,下划线,以及冒号组成,且必须满足正则表达式 [a-zA-Z_:][a-zA-Z0-9_:]*。
  • Lables(标签):使同一个时间序列有了不同维度的识别。例如 http_requests_total{method=“Get”} 表示所有 http 请求中的 Get 请求。当 method=“post” 时,则为新的一个 metric。标签中的键由 ASCII 字符,数字,以及下划线组成,且必须满足正则表达式 [a-zA-Z_:][a-zA-Z0-9_:]*。
  • **timestamp(时间戳)**:数据点的时间,表示数据记录的时间。
  • Sample Value(采样值):实际的时间序列,每个序列包括一个 float64 的值和一个毫秒级的时间戳。

业务侧实现一个接口,返回Prometheus规范化数据

1
2
3
4
5
6
7
8

traefik_entrypoint_requests_total{app="traefik", code="200", entrypoint="metrics", instance="121.21.48.198:9100", job="kubernetes-service-endpoints", method="GET", namespace="default", node="k8s06", protocol="http", service="traefik"}
traefik_entrypoint_requests_total{app="traefik", code="200", entrypoint="websecure", instance="121.21.48.198:9100", job="kubernetes-service-endpoints", method="GET", namespace="default", node="k8s06", protocol="http", service="traefik"}
traefik_entrypoint_requests_total{app="traefik", code="400", entrypoint="websecure", instance="121.21.48.198:9100", job="kubernetes-service-endpoints", method="GET", namespace="default", node="k8s06", protocol="http", service="traefik"}
traefik_entrypoint_requests_total{app="traefik", code="404", entrypoint="traefik", instance="121.21.48.198:9100", job="kubernetes-service-endpoints", method="GET", namespace="default", node="k8s06", protocol="http", service="traefik"}
traefik_entrypoint_requests_total{app="traefik", code="404", entrypoint="web", instance="121.21.48.198:9100", job="kubernetes-service-endpoints", method="GET", namespace="default", node="k8s06", protocol="http", service="traefik"}
traefik_entrypoint_requests_total{app="traefik", code="404", entrypoint="websecure", instance="121.21.48.198:9100", job="kubernetes-service-endpoints", method="GET", namespace="default", node="k8s06", protocol="http", service="traefik"}

运维侧部署的时候,在svc上带上3个标签

1
2
3
4
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: '/monitor'
prometheus.io/port: '9100'
  • prometheus.io/scrape:

  • 自动采集指标数据开关,默认为false;Prometheus会在k8s集群中自动检测哪些svc是打开了这个开关

  • prometheus.io/path:
    采集指标数据路径,默认为 /metrics

  • prometheus.io/port:
    采集指标端口,默认为pod暴露的端口

常见问题

mount

storageClassName: 名称不同匹配不到

Permission denied 远程目录权限

securityContext.fsGroup

1
2
3
4
5
6
securityContext:
runAsUser: 65534
runAsNonRoot: true
runAsGroup: 65534
fsGroup: 65534
fsGroupChangePolicy: "OnRootMismatch" #1.22以上,首次启动时才会执行chmod或chown操作

PVC不能安装两次

Unable to attach or mount volumes: unmounted volumes=[storage-volume], unattached volumes=[kube-api-access-jrz96 storage-volume config-volume]: timed out waiting for the condition

/var/lib/kubelet/pods/eb4c9f58-b7c7-4663-8f9a-1cc447c51fe6/volumes/kubernetes.io~nfs/prometheus-pv0 Output: mount.nfs: mounting 192.168.122.6:/opt/data/k8s/prometheus/pv0 failed, reason given by server: No such file or directory

检查权限

Connection refused

① 首先看nfs服务是否开启
② 其次看rpcbind是否开启

k8s cert manager

cert-manager

它支持从Let’s Encrypt,HashiCorp Vault,Venafi等颁发证书

Issuer/ClusterIssuer

Certificate

ACME Orders and Challenges

Webhook

CA Injector

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

+-------------+
| Ingress/ |
| annotations |
+------+------+
|
| watch ingress change
|
v
+-------------+
| Issuer/ |
| ClusterIssuer |
+------+------+
|
| Create CertificateRequest
|
v
+------+------+
|CertificateRequest|
+------+------+
|
| Create Order
|
v
+------+------+
| Order |
+------+------+
|
| Create Challenges
|
v
+------+------+
| Challenge |
+------+------+
|
| Respond to Challenge
|
v
+------+------+
|ChallengeResponse|
+------+------+
|
| Issue Certificate
|
v
+------+------+
| Secret |
+------+------+

doc cert-manager

https://cert-manager.io/docs/installation/helm/

1
2
3
4
5
6
7
8
9
10
❯ helm repo add jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories
❯ helm pull jetstack/cert-manager --untar
❯ kubectl apply -f ./cert-manager.crds.yaml #--set installCRDs=true
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
1
2
3
4
5
6
7
8
9
helm install -f ./cert-manager/values.yaml \
-name cert-manager \
--namespace cert-manager \
./cert-manager \
--create-namespace \
--version v1.12.4 \
# --set installCRDs=true
--set prometheus.enabled=false \ # Example: disabling prometheus using a Helm parameter
--set webhook.timeoutSeconds=4 # Example: changing the webhook timeout using a Helm parameter
1
2
3
4
5
helm install -f ./cert-manager/values.yaml -name cert-manager --namespace cert-manager  ./cert-manager  --create-namespace --set webhook.timeoutSeconds=4 --set installCRDs=true

❯ helm uninstall -name cert-manager --namespace cert-manager
release "cert-manager" uninstalled

kubectl commands

安装

https://kubernetes.io/zh-cn/docs/tasks/tools/install-kubectl-linux/

kubectl 版本和集群版本之间的差异必须在一个小版本号内。 例如:v1.28 版本的客户端能与 v1.27、 v1.28 和 v1.29 版本的控制面通信。 用最新兼容版的 kubectl 有助于避免不可预见的问题。

shell 自动补全

bash-completion

1
2
3
apt-get install bash-completion 

yum install bash-completion

Bash、Fish 和 Zsh

1
2
3
echo $SHELL  

echo 'source <(kubectl completion bash)' >>~/.bashrc

commands

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs

Dashboard UI

install

https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml

dashboard
kubectl apply -f recommended.yaml
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: v1 kind: Namespace metadata: name: kubernetes-dashboard
---
apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard
---
kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: ports: - port: 443 targetPort: 8443 selector: k8s-app: kubernetes-dashboard
---
apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-certs namespace: kubernetes-dashboard type: Opaque
---
apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-csrf namespace: kubernetes-dashboard type: Opaque data: csrf: ""
---
apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-key-holder namespace: kubernetes-dashboard type: Opaque
---
kind: ConfigMap apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-settings namespace: kubernetes-dashboard
---
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard rules: # Allow Dashboard to get, update and delete Dashboard exclusive secrets. - apiGroups: [""] resources: ["secrets"] resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"] verbs: ["get", "update", "delete"] # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map. - apiGroups: [""] resources: ["configmaps"] resourceNames: ["kubernetes-dashboard-settings"] verbs: ["get", "update"] # Allow Dashboard to get metrics. - apiGroups: [""] resources: ["services"] resourceNames: ["heapster", "dashboard-metrics-scraper"] verbs: ["proxy"] - apiGroups: [""] resources: ["services/proxy"] resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"] verbs: ["get"]
---
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard rules: # Allow Metrics Scraper to get metrics from the Metrics server - apiGroups: ["metrics.k8s.io"] resources: ["pods", "nodes"] verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubernetes-dashboard subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kubernetes-dashboard subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard
---
kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: kubernetes-dashboard template: metadata: labels: k8s-app: kubernetes-dashboard spec: securityContext: seccompProfile: type: RuntimeDefault containers: - name: kubernetes-dashboard image: kubernetesui/dashboard:v2.7.0 imagePullPolicy: Always ports: - containerPort: 8443 protocol: TCP args: - --auto-generate-certificates - --namespace=kubernetes-dashboard # Uncomment the following line to manually specify Kubernetes API server Host # If not specified, Dashboard will attempt to auto discover the API server and connect # to it. Uncomment only if the default does not work. # - --apiserver-host=http://my-address:port volumeMounts: - name: kubernetes-dashboard-certs mountPath: /certs # Create on-disk volume to store exec logs - mountPath: /tmp name: tmp-volume livenessProbe: httpGet: scheme: HTTPS path: / port: 8443 initialDelaySeconds: 30 timeoutSeconds: 30 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 2001 volumes: - name: kubernetes-dashboard-certs secret: secretName: kubernetes-dashboard-certs - name: tmp-volume emptyDir: {} serviceAccountName: kubernetes-dashboard nodeSelector: "kubernetes.io/os": linux # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule
---
kind: Service apiVersion: v1 metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboard spec: ports: - port: 8000 targetPort: 8000 selector: k8s-app: dashboard-metrics-scraper
---
kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboard spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: dashboard-metrics-scraper template: metadata: labels: k8s-app: dashboard-metrics-scraper spec: securityContext: seccompProfile: type: RuntimeDefault containers: - name: dashboard-metrics-scraper image: kubernetesui/metrics-scraper:v1.0.8 ports: - containerPort: 8000 protocol: TCP livenessProbe: httpGet: scheme: HTTP path: / port: 8000 initialDelaySeconds: 30 timeoutSeconds: 30 volumeMounts: - mountPath: /tmp name: tmp-volume securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 2001 serviceAccountName: kubernetes-dashboard nodeSelector: "kubernetes.io/os": linux # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule volumes: - name: tmp-volume emptyDir: {}
1
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml

❯ grep “image: kubernetesui” ./recommended.yaml
image: kubernetesui/dashboard:v2.7.0
image: kubernetesui/metrics-scraper:v1.0.8

载入天数...载入时分秒... ,