vmctl migrate data

vmctl

prometheus

❯ ./vmctl-linux-amd64  prometheus  --help

• –vm-concurrency: 导入数据并发线程。
• –vm-batch-size: 每批大小。
• –prom-concurrency: 同时运行的快照读取器个数。
• –prom-filter-time-start: 选择时间戳大于等于所提供值的时间序列。例如：2020-01-01T20:07:00Z
• –prom-filter-time-end: 选择时间戳小于等于指定值的时间序列。例如：2020-01-01T20:07:00Z
• –prom-filter-label: 过滤标签。例如：‘_name_’ 按名称过滤。
• –prom-filter-label-value: 过滤值的正则，默认 ‘*’。
• –vm-addr: 执行导入请求的VictoriaMetrics地址。导入到集群版本时，不要忘记另外设置–vm-account-id标志。
• –vm-compress: 是否压缩，默认true。
• –vm-rate-limit：数据传输速率，单位：bytes/s，默认（0）关闭。

prometheus.yaml

开启  --web.enable-admin-api
---
version: '3.8'
services:
  prometheus:
    image: prom/prometheus:v2.54.1
    container_name: prometheus
    volumes:
      - "/mnt/lvm/oss/tdb/prometheus:/prometheus:rw"
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.enable-admin-api"
      - "--storage.tsdb.retention.time=3d"
      - "--storage.tsdb.retention.size=1GB"
      - "--storage.tsdb.wal-compression"
    ports:
      - "9090:9090"  

快照请求

❯ curl -XPOST http://localhost:9090/api/v1/admin/tsdb/snapshot
{"status":"success","data":{"name":"20240901T054908Z-791f407899234722"}}

account

❯ curl -u vmselect:vmselect123  https://vmselect.local.org/admin/tenants  -k
{"status":"success","data":["0:0"]}

http://local.org:18480/metrics
vm_tenant_inserted_rows_total{type=”promremotewrite”,accountID=”0”,projectID=”0”} 129585

迁移

❯ sudo ./vmctl-linux-amd64  prometheus  --prom-snapshot /mnt/lvm/oss/tdb/prometheus/snapshots/20240901T054908Z-791f407899234722 \
--vm-concurrency=1 \
--vm-batch-size=200000  \
--prom-concurrency=3 \
--vm-addr http://localhost:18480 \
--vm-account-id 0

[sudo] cs 的密码：
Prometheus import mode
Prometheus snapshot stats:
blocks found: 6;
blocks skipped by time filter: 0;
min time: 1725071505416 (2024-08-31T10:31:45+08:00);
max time: 1725169744452 (2024-09-01T13:49:04+08:00);
samples: 30418386;
series: 26115.
Found 6 blocks to import. Continue? [Y/n] y
VM worker 0:↙ 1463104 samples/s
Processing blocks: 5 / 6 [███████████████████████████████████ █▒▒▒▒▒▒▒] 83.33%
2024/09/01 13:56:52 Import finished!
2024/09/01 13:56:52 VictoriaMetrics importer stats:
idle duration: 1.396944447s;
time spent while importing: 21.02520636s;
total samples: 30418386;
samples/s: 1446758.02;
total bytes: 556.5 MB;
bytes/s: 26.5 MB;
VM worker 0:← 1463104 samples/s
Processing blocks: 6 / 6 [███████████████████████████████████████████] 100.00%
2024/09/01 13:56:52 Total time: 24.472052606s

thanos

s3

https://quay.io/repository/thanos/thanos

https://github.com/G-Research/thanos-remote-read

store-sd.yaml

store-sd.yaml
- targets:
  - thanos-store:10901  

objstore-config.yaml

objstore-config.yaml
#https://thanos.io/tip/thanos/storage.md/#s3 
type: S3
config:
  bucket: "tsdb"
  endpoint: "minio.ui.k8s.cn:9000"
  region: ""
  access_key: "blGDf93r2z7JI7LHpfop"
  insecure: false
  signature_version2: false
  secret_key: "TJUl16aHrjMdFGHx1oId6ZQBDPBSzdhbFWyFRmEN"
  # put_user_metadata: {}
  http_config:
    idle_conn_timeout: 1m30s
    response_header_timeout: 2m
    insecure_skip_verify: true
  trace:
    enable: false
  list_objects_version: ""
  part_size: 134217728  

thanos.yaml

thanos.yaml
version: '3.8'
services:
  thanos-store:
    image: k8s.org/cs/thanos:v0.32.4
    container_name: thanos-store
    volumes:
      - "./objstore-config.yaml:/etc/objstore-config.yaml:ro"
    command:
      - "store"
      - "--data-dir=/tmp/data"
      - "--objstore.config-file=/etc/objstore-config.yaml"
    ports:
      - 10902
      - 10901
    extra_hosts:
       - "minio.ui.k8s.cn:192.168.122.1" # 添加主机名
  thanos-query:
    image: k8s.org/cs/thanos:v0.32.4
    container_name: thanos-query
    volumes:
      - "./store-sd.yaml:/etc/store-sd.yaml:ro"
    command:
      - "query"
      - "--store.sd-files=/etc/store-sd.yaml"
    ports:
      - "10902:10902"
      # - "10901:10901" 
  thanos-remote-read:
    image: thanos-remote-read:latest
    container_name: thanos-remote-read
    command:
      - "-store"
      - "thanos-query:10901"
    ports:
      - "10080:10080"