JMX-Exporter를 통한 prometheus 연동
JMX-Exporter
JMX-Exporter는 Java 에이전트로 동작하며, HTTP서버와 로컬 JVM의 메트릭을 수집한다. 별도의 서버로 존재하며 원격 JVM 타겟의 메트릭을 수집할 수 도 있지만, 프로세스 관련 메트릭(CPU/Memory 등)을 수집하기 어려운 등의 문제가 있어 자바 에이전트로 구동하는 것이 권장된다.
JMX-Exporter 설치 및 기동
Kafka Server Stop
JMX-Exporter를 사용하기 위해서는 Kafka 서버를 정지시켜야 한다.
bin/kafka-server-stop.sh
JMX-Exporter 다운로드
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
Kafka에 JMX-Exporter를 적용하기 위해 환경변수 설정
Kafka 시작 시에 JMX-Exporter를 자바 에이전트로 함께 기동하기 위해 KAFKAOPTS에 JMX-Exporter의 컨피그 파일과 .jar위치, Prometheus가 수집할 포트 번호를 환경변수로 만든다.
export KAFKA_OPTS="-javaagent:/kafka/kafka_2.12-2.4.0/jmx/jmx_prometheus_javaagent-0.12.0.jar=7071:/kafka/kafka_2.12-2.4.0/jmx/kafka_config.yml"
JMX-Exporter Config 파일 작성
lowercaseOutputName: true rules: # Special cases and very specific rules - pattern : kafka.server<>Value name: kafka_server_$1_$2 type: GAUGE labels: clientId: "$3" topic: "$4" partition: "$5" - pattern : kafka.server <>Value name: kafka_server_$1_$2 type: GAUGE labels: clientId: "$3" broker: "$4:$5" - pattern : kafka.coordinator.(\w+) <>Value name: kafka_coordinator_$1_$2_$3 type: GAUGE # Generic per-second counters with 0-2 key/value pairs - pattern: kafka.(\w+) <>Count name: kafka_$1_$2_$3_total type: COUNTER labels: "$4": "$5" "$6": "$7" - pattern: kafka.(\w+) <>Count name: kafka_$1_$2_$3_total type: COUNTER labels: "$4": "$5" - pattern: kafka.(\w+) <>Count name: kafka_$1_$2_$3_total type: COUNTER - pattern: kafka.server <>([a-z-]+) name: kafka_server_quota_$3 type: GAUGE labels: resource: "$1" clientId: "$2" - pattern: kafka.server <>([a-z-]+) name: kafka_server_quota_$4 type: GAUGE labels: resource: "$1" user: "$2" clientId: "$3" # Generic gauges with 0-2 key/value pairs - pattern: kafka.(\w+) <>Value name: kafka_$1_$2_$3 type: GAUGE labels: "$4": "$5" "$6": "$7" - pattern: kafka.(\w+) <>Value name: kafka_$1_$2_$3 type: GAUGE labels: "$4": "$5" - pattern: kafka.(\w+) <>Value name: kafka_$1_$2_$3 type: GAUGE # Emulate Prometheus 'Summary' metrics for the exported 'Histogram's. # # Note that these are missing the '_sum' metric! - pattern: kafka.(\w+) <>Count name: kafka_$1_$2_$3_count type: COUNTER labels: "$4": "$5" "$6": "$7" - pattern: kafka.(\w+) <>(\d+)thPercentile name: kafka_$1_$2_$3 type: GAUGE labels: "$4": "$5" "$6": "$7" quantile: "0.$8" - pattern: kafka.(\w+) <>Count name: kafka_$1_$2_$3_count type: COUNTER labels: "$4": "$5" - pattern: kafka.(\w+) <>(\d+)thPercentile name: kafka_$1_$2_$3 type: GAUGE labels: "$4": "$5" quantile: "0.$6" - pattern: kafka.(\w+) <>Count name: kafka_$1_$2_$3_count type: COUNTER - pattern: kafka.(\w+) <>(\d+)thPercentile name: kafka_$1_$2_$3 type: GAUGE labels: quantile: "0.$4"
Kafka 서버 재기동
nohup bin/kafka-server-start.sh config/server.properties &
서버 메트릭 확인
curl http://localhost:7071/metrics ... # TYPE jmx_scrape_error gauge jmx_scrape_error 0.0 # HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool. # TYPE jvm_buffer_pool_used_bytes gauge jvm_buffer_pool_used_bytes{pool="direct",} 5641087.0 jvm_buffer_pool_used_bytes{pool="mapped",} 4.1943032E8 # HELP jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool. # TYPE jvm_buffer_pool_capacity_bytes gauge jvm_buffer_pool_capacity_bytes{pool="direct",} 5641087.0 jvm_buffer_pool_capacity_bytes{pool="mapped",} 4.1943032E8 # HELP jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool. # TYPE jvm_buffer_pool_used_buffers gauge jvm_buffer_pool_used_buffers{pool="direct",} 19.0 jvm_buffer_pool_used_buffers{pool="mapped",} 40.0 ...
Prometheus 연동
Prometheus 컨피그 파일 추가
scrape_configs: # The job name is added as a label `job=` to any timeseries scraped from this config. - job_name: kafka static_configs: - targets: ['192.168.150.75:7071'] # kafka labels: instance : kafka
Prometheus 재기동
/root/prometheus/prometheus --config.file=/root/prometheus/prometheus.yml