logstash监控

发表于   |   更新于 | 分类于 | | 阅读次数: | 总览

logstash api 监控

        Logstash 5.0 开始，提供了输出自身进程的指标和状态监控的 API。这大大降低了我们监控 Logstash 的难度。可以使用Logstash提供的监视API来检索这些指标。默认情况下api可用，不需任何额外的配置，或者可以配置X-Pack监视以将数据发送到监视群集。这个直接在kibana页面进行配置。

        Logstash收集的指标包括:

  • 节点信息: 接口目前支持三种类型：pipeline、os、jvm。
  • 插件信息: 包括已安装插件的列表。
  • 节点统计指标: JVM统计信息，进程统计信息，与事件相关的统计信息和管道运行时统计信息。
  • 热线程统计

        检索常规信息

1
2
3
4
5
6
7
8
9
10
11
$ curl -XGET 'localhost:9600/?pretty'
{
  "host" : "prod-elk-logstash-02",
  "version" : "6.4.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "57c08022-8064-4bce-8c9b-e6db6e005479",
  "name" : "prod-elk-logstash-02",
  "build_date" : "2018-08-18T00:25:22Z",
  "build_sha" : "f8014ac54e6c8ff6c071c0960ca1b00e9735f43a",
  "build_snapshot" : false
}

pipeline 指标

工作程序数量，批处理大小和批处理延迟

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ curl -XGET 'localhost:9600/_node/pipelines?pretty'
{
  "host" : "prod-elk-logstash-02",
  "version" : "6.4.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "57c08022-8064-4bce-8c9b-e6db6e005479",
  "name" : "prod-elk-logstash-02",
  "pipelines" : {
    "main" : {
      "workers" : 4,
      "batch_size" : 125,
      "batch_delay" : 50,
      "config_reload_automatic" : false,
      "config_reload_interval" : 3000000000,
      "dead_letter_queue_enabled" : false
    }
  }
}

特定管道的信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ curl -XGET 'localhost:9600/_node/pipelines/main?pretty'
{
  "host" : "prod-elk-logstash-02",
  "version" : "6.4.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "57c08022-8064-4bce-8c9b-e6db6e005479",
  "name" : "prod-elk-logstash-02",
  "pipelines" : {
    "main" : {
      "workers" : 4,
      "batch_size" : 125,
      "batch_delay" : 50,
      "config_reload_automatic" : false,
      "config_reload_interval" : 3000000000,
      "dead_letter_queue_enabled" : false
    }
  }
}

OS 信息

显示了操作系统名称，体系结构，版本和可用处理器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ curl -XGET 'localhost:9600/_node/os?pretty'
{
  "host" : "prod-elk-logstash-02",
  "version" : "6.4.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "57c08022-8064-4bce-8c9b-e6db6e005479",
  "name" : "prod-elk-logstash-02",
  "os" : {
    "name" : "Linux",
    "arch" : "amd64",
    "version" : "3.10.0-693.11.1.el7.x86_64",
    "available_processors" : 4
  }
}

jvm 信息

        显示节点级别的JVM统计信息，例如JVM进程ID，版本，VM信息，内存使用情况以及有关垃圾收集器的信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ curl -XGET 'localhost:9600/_node/jvm?pretty'
{
  "host" : "prod-elk-logstash-02",
  "version" : "6.4.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "57c08022-8064-4bce-8c9b-e6db6e005479",
  "name" : "prod-elk-logstash-02",
  "jvm" : {
    "pid" : 20827,
    "version" : "1.8.0_144",
    "vm_version" : "1.8.0_144",
    "vm_vendor" : "Oracle Corporation",
    "vm_name" : "Java HotSpot(TM) 64-Bit Server VM",
    "start_time_in_millis" : 1574066157917,
    "mem" : {
      "heap_init_in_bytes" : 3221225472,
      "heap_max_in_bytes" : 3186360320,
      "non_heap_init_in_bytes" : 2555904,
      "non_heap_max_in_bytes" : 0
    },
    "gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ]
  }
}

插件信息

        插件信息API获取有关当前安装的所有Logstash插件的信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ curl -XGET 'localhost:9600/_node/plugins?pretty'
{
  "host" : "prod-elk-logstash-02",
  "version" : "6.4.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "57c08022-8064-4bce-8c9b-e6db6e005479",
  "name" : "prod-elk-logstash-02",
  "total" : 98,
  "plugins" : [ {
    "name" : "logstash-codec-cef",
    "version" : "5.0.3"
  }, {
    "name" : "logstash-codec-collectd",
    "version" : "3.0.8"
  }, {
    "name" : "logstash-codec-dots",
    "version" : "3.0.6"
  }, {
    "name" : "logstash-codec-edn",
    "version" : "3.0.6"
  },……………………
}

节点统计指表

        节点统计信息API检索有关Logstash的运行时统计信息。

jvm统计

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
$ curl -XGET 'localhost:9600/_node/stats/jvm?pretty'
{
  "host" : "prod-elk-logstash-02",
  "version" : "6.4.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "57c08022-8064-4bce-8c9b-e6db6e005479",
  "name" : "prod-elk-logstash-02",
  "jvm" : {
    "threads" : {
      "count" : 55,
      "peak_count" : 56
    },
    "mem" : {
      "heap_used_percent" : 10,
      "heap_committed_in_bytes" : 3186360320,
      "heap_max_in_bytes" : 3186360320,
      "heap_used_in_bytes" : 348033216,
      "non_heap_used_in_bytes" : 170512848,
      "non_heap_committed_in_bytes" : 201850880,
      "pools" : {
        "survivor" : {
          "peak_used_in_bytes" : 34865152,
          "used_in_bytes" : 1629704,
          "peak_max_in_bytes" : 34865152,
          "max_in_bytes" : 34865152,
          "committed_in_bytes" : 34865152
        },
        "old" : {
          "peak_used_in_bytes" : 230612312,
          "used_in_bytes" : 230612312,
          "peak_max_in_bytes" : 2872311808,
          "max_in_bytes" : 2872311808,
          "committed_in_bytes" : 2872311808
        },
        "young" : {
          "peak_used_in_bytes" : 279183360,
          "used_in_bytes" : 115791200,
          "peak_max_in_bytes" : 279183360,
          "max_in_bytes" : 279183360,
          "committed_in_bytes" : 279183360
        }
      }
    },
    "gc" : {
      "collectors" : {
        "old" : {
          "collection_time_in_millis" : 301,
          "collection_count" : 2
        },
        "young" : {
          "collection_time_in_millis" : 2536,
          "collection_count" : 98
        }
      }
    },
    "uptime_in_millis" : 66149287
  }
}

进程统计

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ curl -XGET 'localhost:9600/_node/stats/process?pretty'
{
  "host" : "prod-elk-logstash-02",
  "version" : "6.4.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "57c08022-8064-4bce-8c9b-e6db6e005479",
  "name" : "prod-elk-logstash-02",
  "process" : {
    "open_file_descriptors" : 169,
    "peak_open_file_descriptors" : 170,
    "max_file_descriptors" : 16384,
    "mem" : {
      "total_virtual_in_bytes" : 7337734144
    },
    "cpu" : {
      "total_in_millis" : 593350,
      "percent" : 0,
      "load_average" : {
        "1m" : 0.01,
        "5m" : 0.07,
        "15m" : 0.1
      }
    }
  }
}

events

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ curl -s localhost:9600/_node/stats/events?pretty=true
{ 
  "host" : "prod-elk-logstash-02",
  "version" : "6.4.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "57c08022-8064-4bce-8c9b-e6db6e005479",
  "name" : "prod-elk-logstash-02",
  "events" : {
    "in" : 103628,
    "filtered" : 103628,
    "out" : 103628,
    "duration_in_millis" : 96810,
    "queue_push_duration_in_millis" : 5104
  }
}

管道统计

  • 每个管道输入，过滤或输出的事件数
  • 每个已配置的滤波器或输出级的统计信息
  • 有关配置重新加载成功和失败的信息（启用配置重新加载时）
  • 关于持久性队列信息（当 永久队列被启用）
1
2
3
4
$ curl -XGET 'localhost:9600/_node/stats/pipelines?pretty'
{
………………
}

        通过包含管道ID来查看特定管道的统计信息

1
2
3
4
$ curl -XGET 'localhost:9600/_node/stats/pipelines/main?pretty'
{
………………
}

刷新统计

        显示有关配置重新加载成功和失败的信息。

1
2
3
4
5
6
7
8
9
10
11
12
$ curl -XGET 'localhost:9600/_node/stats/reloads?pretty'
{
  "host" : "prod-elk-logstash-02",
  "version" : "6.4.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "57c08022-8064-4bce-8c9b-e6db6e005479",
  "name" : "prod-elk-logstash-02",
  "reloads" : {
    "successes" : 0,
    "failures" : 0
  }
}

os 统计

        当Logstash在容器中运行时，请求将返回一个包含cgroup信息的JSON文档，提供CPU负载的更准确视图，包括是否对容器进行了限制。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ curl -XGET 'localhost:9600/_node/stats/os?pretty'
{
  "host" : "prod-elk-logstash-02",
  "version" : "6.4.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "57c08022-8064-4bce-8c9b-e6db6e005479",
  "name" : "prod-elk-logstash-02",
  "os" : {
    "cgroup" : {
      "cpuacct" : {
        "usage_nanos" : 20928023701495776,
        "control_group" : "/"
      },
      "cpu" : {
        "cfs_quota_micros" : -1,
        "control_group" : "/",
        "stat" : {
          "number_of_times_throttled" : 0,
          "time_throttled_nanos" : 0,
          "number_of_elapsed_periods" : 0
        },
        "cfs_period_micros" : 100000
      }
    }
  }
}

热线程api

        热线程API获取Logstash的当前热线程。热线程是Java线程，具有很高的CPU使用率，并且执行时间比正常时间长。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
$ curl -XGET 'localhost:9600/_node/hot_threads?pretty'
{
  "host" : "prod-elk-logstash-02",
  "version" : "6.4.0",
  "http_address" : "127.0.0.1:9600",
  "id" : "57c08022-8064-4bce-8c9b-e6db6e005479",
  "name" : "prod-elk-logstash-02",
  "hot_threads" : {
    "time" : "2019-11-19T11:07:05+08:00",
    "busiest_threads" : 3,
    "threads" : [ {
      "name" : "Ruby-0-Thread-23@[main]>worker1",
      "thread_id" : 42,
      "percent_of_cpu_time" : 0.1,
      "state" : "timed_waiting",
      "path" : ":1",
      "traces" : [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)" ]
    }, {
      "name" : "Ruby-0-Thread-24@[main]>worker2",
      "thread_id" : 43,
      "percent_of_cpu_time" : 0.06,
      "state" : "timed_waiting",
      "path" : ":1",
      "traces" : [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)" ]
    }, {
      "name" : "Ruby-0-Thread-22@[main]>worker0",
      "thread_id" : 41,
      "percent_of_cpu_time" : 0.06,
      "state" : "timed_waiting",
      "path" : ":1",
      "traces" : [ "sun.misc.Unsafe.park(Native Method)", "java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)", "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)" ]
    } ]
  }
}

可执行的参数:

  • threads: 返回的热线程数。预设值为10。
  • stacktrace_size: 要为每个线程报告的堆栈跟踪的深度。默认值为50。
  • human: 如果为true，则返回纯文本而不是JSON格式。默认为false。
  • ignore_idle_threads: 如果为true，则不返回空闲线程。默认值为true。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ curl -XGET 'localhost:9600/_node/hot_threads?human=true'
Hot threads at 2019-11-19T11:07:43+08:00, busiestThreads=3:
================================================================================
0.1 % of cpu usage, state: timed_waiting, thread name: 'Ruby-0-Thread-23@[main]>worker1', thread id: 42
:1
        sun.misc.Unsafe.park(Native Method)
        java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
--------------------------------------------------------------------------------
0.06 % of cpu usage, state: timed_waiting, thread name: 'Ruby-0-Thread-24@[main]>worker2', thread id: 43
:1
        sun.misc.Unsafe.park(Native Method)
        java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
--------------------------------------------------------------------------------
0.06 % of cpu usage, state: timed_waiting, thread name: 'Ruby-0-Thread-22@[main]>worker0', thread id: 41
:1
        sun.misc.Unsafe.park(Native Method)
        java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
--------------------------------------------------------------------------------

        上面的一些api监控，可以写成脚本的形式，使用zabbix来进行监控。除了使用zabbix来监控logstash外，我们还可以使用自带的x-pack来进行监控，介绍可以参考官方。

X-Pack监视概述

        修改logstash.yml配置文件，在配置文件里面只需要修改两个参数即可 xpack.monitoring.enabled 、xpack.monitoring.elasticsearch.url。监控Elasticsearch的是启用 xpack.monitoring.collection.enabled

修改logstash.yml

1
2
3
$ cat logstash.yml
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: ["http://es1:9200","http://es2:9200","http://es3:9200"]

重启logstash

1
$ sudo  systemctl  restart logstash

kibana 查看

        登录kibana进行查看
img

点击node查看
img

这里顺道点考kibana
img

坚持原创技术分享，您的支持将鼓励我继续创作！
0%