Elasticsearch索引管理

发表于   |   更新于 | 分类于 | | 阅读次数: | 总览

痛点

        公司上线了elk日志分析系统，但是在线下开发测试环境数据量比较大，而且线下都是一些测试和开发使用的数据。也没有必要存放起来。但是随着时间的推移，elasticsearch的数据量会撑爆磁盘。伴随着需要人为的清理。所以这就要删除一定时间的数据，来保障磁盘空间的使用量。
        由于索引过多，之前都是实用脚本来管理，比如：增量rollover动态更新脚本、定期delete脚本、定期force_merge脚本、定期shrink脚本、定期快照脚本，随着需求的过多，es也有三个集群，脚本维护虽然用了一定的科学管理办法，但是还是挺难维护的。

elasticsearch-curator介绍

        elasticsearch-curator管理elasticsearch索引和快照，从集群里获取全部索引或者快照作为可操作列表；迭代用户定义的过滤器列表，根据需要逐步从此可操作列表中删除索引或快照；对保留下来的列表执行各种操作。

功能

        curator允许对索引和快照执行许多不同的操作，包括：

  • 从别名添加或删除索引（或两者！）
  • 更改分片路由分配更改分片路由分配
  • 关闭索引关闭索引
  • 建索引创建索引
  • 删除索引删除索引
  • 删除快照删除快照
  • 打开被关闭的索引打开被关闭的索引
  • 对索引执行forcemerge段合并操作对索引执行forcemerge段合并操作
  • ndex索引，包括来自远程集群的索引reindex索引，包括来自远程集群的索引
  • 更改索引的每个分片的副本数 更改索引的每个分片的副本数
  • rollover索引rollover索引
  • 生成索引的快照（备份）生成索引的快照（备份）
  • 还原快照还原快照

elasticsearch-curator安装

        elasticsearch-curator安装可以通过两种方式来进行安装，一种是pip、一种是yum。这里使用pip来进行安装。安装参考

pip 安装elasticsearch-curator

1
# pip install elasticsearch-curator

验证

1
2
3
# curator_cli --host 127.0.0.1 --port 9200  show_indices

# 如果es安装监听端口是0.0.0.0，则host参数和port参数不需要

配置 config.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# mkdir -p /opt/ELKStack/curator

# cat > config.yaml<<EOF
client:
  hosts: [ "172.21.16.18", "172.21.16.19", "172.21.16.20" ]
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']
EOF
  • CONFIG.YML是配置文件，用于配置ES集群信息，官方参考
    • 集群IP
    • 安全认证信息
    • 日志信息

配置action.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# cat >action.yml <<EOF
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 30 days (based on index name), for logstash-prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      # 保留 kibana|json|monitoring|metadata 不被清理
      value: '^((?!(kibana|json|monitoring|metadata)).)*$'
    - filtertype: age
      source: creation_date
      direction: older
      #timestring: '%Yi-%m-%d'
      unit: days
      unit_count: 30
EOF

        如果执行多个任务，在actions: 后面的依次类推，官方参考

执行测试

1
# /usr/bin/curator --config /opt/ELKStack/curator/config.yaml /opt/ELKStack/curator/action.yml 1>> /tmp/curator.log 2>&1

  • 执行前

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    # curator_cli --host 127.0.0.1 --port 9200  show_indices
    2019-12-30 14:38:22,438 INFO      Instantiating client object
    2019-12-30 14:38:22,439 INFO      Testing client connectivity
    2019-12-30 14:38:22,506 INFO      Successfully created Elasticsearch client object with provided settings
    .kibana6.4
    .monitoring-es-6-2019.12.23
    .monitoring-es-6-2019.12.24
    .monitoring-es-6-2019.12.25
    .monitoring-es-6-2019.12.26
    .monitoring-es-6-2019.12.27
    .monitoring-es-6-2019.12.28
    .monitoring-es-6-2019.12.29
    .monitoring-es-6-2019.12.30
    .monitoring-kibana-6-2019.12.23
    .monitoring-kibana-6-2019.12.24
    .monitoring-kibana-6-2019.12.25
    .monitoring-kibana-6-2019.12.26
    .monitoring-kibana-6-2019.12.27
    .monitoring-kibana-6-2019.12.28
    .monitoring-kibana-6-2019.12.29
    .monitoring-kibana-6-2019.12.30
    uat-xxx-xxx-system-2019-11
    uat-xxx-xxx-system-2019-12
    uat-xxx-xxx-system-2019-11
    uat-xxx-xxx-system-2019-12
    uat-xxx-xxx-xxx-system-2019-11
    uat-xxx-xxx-xxx-system-2019-12
    dev-xxx-xxx-system-2019-12
    dev-xxx-xxx-system-2019-11
    dev-xxx-xxx-system-2019-12
    dev-xxx-xxx-system-2019-11
    dev-xxx-xxx-system-2019-12
    dev-xxx-xxx-xxx-system-2019-11
    dev-xxx-xxx-xxx-system-2019-12
    test-xxx-xxx-system-2019-11
    test-xxx-xxx-system-2019-12
    test-xxx-xxx-xxx-system-2019-11
    test-xxx-xxx-xxx-system-2019-12
    zxc-2019-11
    zxc-2019-12

  • 执行后

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    # cat /tmp/curator.log 
    2019-12-30 16:09:44,383 INFO      Preparing Action ID: 1, "delete_indices"
    2019-12-30 16:09:44,384 INFO      Creating client object and testing connection
    2019-12-30 16:09:44,387 INFO      Instantiating client object
    2019-12-30 16:09:44,389 INFO      Testing client connectivity
    2019-12-30 16:09:44,398 INFO      Successfully created Elasticsearch client object with provided settings
    2019-12-30 16:09:44,403 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 30 days. Ignore the error if the    filter does not result in an actionable list of indices    (ignore_empty_list) and exit cleanly.
    2019-12-30 16:09:46,667 INFO      Deleting 9 selected indices: [u'dev-xxx-xxx-system-2019-11', u'uat-xxx-xxx-system-2019-11', u'test-xxx-xxx-xxx-system-2019-11', u'test-xxx-xxx-system-2019-11', u'uat-xxx-xxx-xxx-system-2019-11', u'dev-xxx-xxx-xxx-system-2019-11', u'uat-xxx-xxx-system-2019-11', u'dev-xxx-xxx-system-2019-11', u'zxc-2019-11']
    2019-12-30 16:09:46,667 INFO      ---deleting index dev-xxx-xxx-system-2019-11
    2019-12-30 16:09:46,667 INFO      ---deleting index uat-xxx-xxx-system-2019-11
    2019-12-30 16:09:46,667 INFO      ---deleting index test-xxx-xxx-xxx-system-2019-11
    2019-12-30 16:09:46,667 INFO      ---deleting index test-xxx-xxx-system-2019-11
    2019-12-30 16:09:46,667 INFO      ---deleting index uat-xxx-xxx-xxx-system-2019-11
    2019-12-30 16:09:46,667 INFO      ---deleting index dev-xxx-xxx-xxx-system-2019-11
    2019-12-30 16:09:46,667 INFO      ---deleting index uat-xxx-xxx-system-2019-11
    2019-12-30 16:09:46,667 INFO      ---deleting index dev-xxx-xxx-system-2019-11
    2019-12-30 16:09:46,667 INFO      ---deleting index zxc-2019-11
    2019-12-30 16:09:56,702 INFO      Action ID: 1, "delete_indices" completed.
    2019-12-30 16:09:56,702 INFO      Job completed.

    # curator_cli --host 127.0.0.1 --port 9200  show_indices
    2019-12-30 16:13:09,174 INFO      Instantiating client object
    2019-12-30 16:13:09,175 INFO      Testing client connectivity
    2019-12-30 16:13:09,182 INFO      Successfully created Elasticsearch client object with provided settings
    .kibana6.4
    .monitoring-es-6-2019.12.23
    .monitoring-es-6-2019.12.24
    .monitoring-es-6-2019.12.25
    .monitoring-es-6-2019.12.26
    .monitoring-es-6-2019.12.27
    .monitoring-es-6-2019.12.28
    .monitoring-es-6-2019.12.29
    .monitoring-es-6-2019.12.30
    .monitoring-kibana-6-2019.12.23
    .monitoring-kibana-6-2019.12.24
    .monitoring-kibana-6-2019.12.25
    .monitoring-kibana-6-2019.12.26
    .monitoring-kibana-6-2019.12.27
    .monitoring-kibana-6-2019.12.28
    .monitoring-kibana-6-2019.12.29
    .monitoring-kibana-6-2019.12.30
    uat-xxx-xxx-system-2019-12
    uat-xxx-xxx-system-2019-12
    uat-xxx-xxx-xxx-system-2019-12
    dev-xxx-xxx-system-2019-12
    dev-xxx-xxx-system-2019-12
    dev-xxx-xxx-system-2019-12
    dev-xxx-xxx-xxx-system-2019-12
    test-xxx-xxx-system-2019-12
    test-xxx-xxx-xxx-system-2019-12
    zxc-2019-12

设置计划任务

1
2
3
4
# crontab -e
42 4 1 * * /usr/bin/curator --config /opt/ELKStack/curator/config.yaml /opt/ELKStack/curator/action.yml 1>> /tmp/curator.log 2>&1

# systemctl restart crond
坚持原创技术分享，您的支持将鼓励我继续创作！
0%