k8s storage

k8s 存储制备器

实现方式

https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner

nfs

kubernetes集群所有节点需要安装nfs客户端

https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner

helm install -f ./nfs-subdir-external-provisioner/values.yaml  ./nfs-subdir-external-provisioner \
-n nfs-provisioner \
 --set storageClass.name=nfs-client \
  --set storageClass.defaultClass=false \
   --set nfs.server=k8s.org  \
   --set nfs.path=/mnt/ssd/data/nfs-storage \
--generate-name

主动指定名称，或者增加 --generate-name 参数让它自动生成一个随机的名称

❯ kubectl create ns nfs-provisioner

通过该配置自动创建

–set global.storageClass=nfs-client

删除pod后如何重新挂载原来的卷

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  pathPattern: "${.PVC.namespace}/${.PVC.annotations.nfs.io/storage-path}" # waits for nfs.io/storage-path annotation, if not specified will accept as empty string.
  onDelete: delete

使用pathPattern路径添加注解

persistence
  enabled: true
  annotations:
      nfs.io/storage-path: "prometheus/name"

yaml

pvc

PersistentVolumeClaim

PersistentVolumeClaim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
{{- if .Values.server.persistentVolume.annotations }}
annotations:
{{ toYaml .Values.server.persistentVolume.annotations | indent 4 }}
{{- end }}


....  

volumeClaimTemplates

volumeClaimTemplates

volumeClaimTemplates
volumeClaimTemplates:
    - metadata:
     name: storage
    {{- with .Values.persistence.annotations }}
     annotations:
    {{- toYaml . | nindent 10 }}
    {{- end }}
    spec:
    accessModes:


.....  

persistence:
  enabled: true
  existingClaim: ""
  accessModes:
    - ReadWriteOnce
  size: 1Gi
  storageClass: nfs-client
  annotations:
     nfs.io/storage-path: "data-kafka-controller"

容器启动提示对挂载目录没有权限

❯  mkdir -p /mnt/ssd/LOCAL_NFS/middleware/data-zookeeper-{0..2}
❯ chmod -R 777 /mnt/ssd/LOCAL_NFS/middleware
'/mnt/ssd/LOCAL_NFS/middleware' 的模式已由 0755 (rwxr-xr-x) 更改为 0777 (rwxrwxrwx)
'/mnt/ssd/LOCAL_NFS/middleware/data-zookeeper-0' 的模式已由 0755 (rwxr-xr-x) 更改为 0777 (rwxrwxrwx)
'/mnt/ssd/LOCAL_NFS/middleware/data-zookeeper-2' 的模式已由 0755 (rwxr-xr-x) 更改为 0777 (rwxrwxrwx)
'/mnt/ssd/LOCAL_NFS/middleware/data-zookeeper-1' 的模式已由 0755 (rwxr-xr-x) 更改为 0777 (rwxrwxrwx)

csi

https://kubernetes-csi.github.io/docs/external-provisioner.html

CSI Driver = Node(DaemonSet) + Controller(StatefuleSet)

• 橙色部分：Identity、Node、Controller 是需要开发者自己实现的，被称为 Custom Components。
• 蓝色部分：node-driver-registrar、external-attacher、external-provisioner 组件是 Kubernetes 团队开发和维护的，被称为 External Components，它们都是以 sidecar 的形式与 Custom Components 配合使用的。

交互	过程
External Provisioner + Controller Server	创建、删除数据卷
External Attacher + Controller Server	执行数据卷的挂载、卸载操作
Volume Manager + Volume Plugin + Node Server	执行数据卷的Mount、Umount操作
AD Controller + VolumePlugin	创建、删除VolumeAttachment对象
External Resizer + Controller Server	执行数据卷的扩容操作
ExternalSnapshotter+ControllerServer	执行数据卷的备份操作
Driver Registrar + VolumeManager + Node Server	注册CSI插件，创建CSINode对象

• controller 根据多备份情况部署，可部署多个，主要负责 provision(动态create/delete) 和 attach(mount/umount)工作

• node driver registrar 注册功能，每个节点部署，它会在每个节点上进行注册

CSI Node 和 CSI Identity 通常是部署在一个容器里面的,他们和 CSI Controller 一起完成 volume 的 mount 操作

https://blog.csdn.net/2301_76975791/article/details/131098408

s3-csi

https://github.com/ctrox/csi-s3

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: csi-s3-existing-bucket
provisioner: ch.ctrox.csi.s3-driver
reclaimPolicy: Retain
parameters:
  mounter: rclone
  bucket: some-existing-bucket-name
  # 'usePrefix' must be true in order to enable the prefix feature and to avoid the removal of the prefix or bucket
  usePrefix: "true"
  # 'prefix' can be empty (it will mount on the root of the bucket), an existing prefix or a new one.
  prefix: custom-prefix

mounter

把S3当文件挂载使用

• rclone
• s3fs
• goofys
• s3backer

rclone-minio

https://rclone.org/s3/#liara-cloud

~/.config/rclone/rclone.conf

[minio]
type = s3
provider = Minio
env_auth = false
access_key_id = USWUXHGYZQYFYFFIT3RE
secret_access_key = MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
region = us-east-1
endpoint = http://192.168.1.106:9000
location_constraint =
server_side_encryption =
[minionew]
type = s3
provider = Minio
env_auth = false
access_key_id = USWUXHGYZQYFYFFIT3RE
secret_access_key = MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
region = us-east-1
endpoint = http://xxxxx:9000
location_constraint =
server_side_encryption =

rclone迁移

前提，两台机器的时区及时间要保持一致

rclone sync minio:bucket minionew:bucket

bucket桶名需要保持一致