JMX-Exporter를 통한 prometheus 연동

JMX-Exporter

JMX-Exporter는 Java 에이전트로 동작하며, HTTP서버와 로컬 JVM의 메트릭을 수집한다. 별도의 서버로 존재하며 원격 JVM 타겟의 메트릭을 수집할 수 도 있지만, 프로세스 관련 메트릭(CPU/Memory 등)을 수집하기 어려운 등의 문제가 있어 자바 에이전트로 구동하는 것이 권장된다.

JMX-Exporter 설치 및 기동

Kafka Server Stop

JMX-Exporter를 사용하기 위해서는 Kafka 서버를 정지시켜야 한다.

bin/kafka-server-stop.sh

JMX-Exporter 다운로드

wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar

Kafka에 JMX-Exporter를 적용하기 위해 환경변수 설정

Kafka 시작 시에 JMX-Exporter를 자바 에이전트로 함께 기동하기 위해 KAFKAOPTS에 JMX-Exporter의 컨피그 파일과 .jar위치, Prometheus가 수집할 포트 번호를 환경변수로 만든다.

export KAFKA_OPTS="-javaagent:/kafka/kafka_2.12-2.4.0/jmx/jmx_prometheus_javaagent-0.12.0.jar=7071:/kafka/kafka_2.12-2.4.0/jmx/kafka_config.yml"

JMX-Exporter Config 파일 작성

lowercaseOutputName: true
 
rules:
# Special cases and very specific rules
- pattern : kafka.server<>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    topic: "$4"
    partition: "$5"
- pattern : kafka.server<>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    broker: "$4:$5"
- pattern : kafka.coordinator.(\w+)<>Value
  name: kafka_coordinator_$1_$2_$3
  type: GAUGE
 
# Generic per-second counters with 0-2 key/value pairs
- pattern: kafka.(\w+)<>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(\w+)<>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER
  labels:
    "$4": "$5"
- pattern: kafka.(\w+)<>Count
  name: kafka_$1_$2_$3_total
  type: COUNTER
 
- pattern: kafka.server<>([a-z-]+)
  name: kafka_server_quota_$3
  type: GAUGE
  labels:
    resource: "$1"
    clientId: "$2"
 
- pattern: kafka.server<>([a-z-]+)
  name: kafka_server_quota_$4
  type: GAUGE
  labels:
    resource: "$1"
    user: "$2"
    clientId: "$3"
 
# Generic gauges with 0-2 key/value pairs
- pattern: kafka.(\w+)<>Value
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(\w+)<>Value
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
- pattern: kafka.(\w+)<>Value
  name: kafka_$1_$2_$3
  type: GAUGE
 
# Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
#
# Note that these are missing the '_sum' metric!
- pattern: kafka.(\w+)<>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
  labels:
    "$4": "$5"
    "$6": "$7"
- pattern: kafka.(\w+)<>(\d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    "$6": "$7"
    quantile: "0.$8"
- pattern: kafka.(\w+)<>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
  labels:
    "$4": "$5"
- pattern: kafka.(\w+)<>(\d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    "$4": "$5"
    quantile: "0.$6"
- pattern: kafka.(\w+)<>Count
  name: kafka_$1_$2_$3_count
  type: COUNTER
- pattern: kafka.(\w+)<>(\d+)thPercentile
  name: kafka_$1_$2_$3
  type: GAUGE
  labels:
    quantile: "0.$4"

Kafka 서버 재기동

nohup bin/kafka-server-start.sh config/server.properties &

서버 메트릭 확인

curl http://localhost:7071/metrics
 
...
# TYPE jmx_scrape_error gauge
jmx_scrape_error 0.0
# HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_bytes gauge
jvm_buffer_pool_used_bytes{pool="direct",} 5641087.0
jvm_buffer_pool_used_bytes{pool="mapped",} 4.1943032E8
# HELP jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool.
# TYPE jvm_buffer_pool_capacity_bytes gauge
jvm_buffer_pool_capacity_bytes{pool="direct",} 5641087.0
jvm_buffer_pool_capacity_bytes{pool="mapped",} 4.1943032E8
# HELP jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_buffers gauge
jvm_buffer_pool_used_buffers{pool="direct",} 19.0
jvm_buffer_pool_used_buffers{pool="mapped",} 40.0
...

Prometheus 연동

Prometheus 컨피그 파일 추가

scrape_configs:
# The job name is added as a label `job=` to any timeseries scraped from this config.
- job_name: kafka
static_configs:
- targets: ['192.168.150.75:7071'] # kafka
labels:
instance : kafka

Prometheus 재기동

/root/prometheus/prometheus --config.file=/root/prometheus/prometheus.yml

Kafka-Prometheus 연동