Quantcast
Channel: 容器云计算,Devops,DBA,网络安全。
Viewing all articles
Browse latest Browse all 780

filebeat+kafka+logstash收集日志到es使用kibana展示

$
0
0

filebeat+kafka+logstash收集日志到es使用kibana展示

实现逻辑

filebeat    ==>>    kafka    <<==    logstash    ==>>    elastsearch   <==    kibana

 

1. filebeat配置

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/messages
  fields:
    log_topic: test_kafka
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
output.kafka:
  enable: true
  hosts: ["log1:9092"]
  version: "2.0.0"
  topic: '%{[fields.log_topic]}'
  partition.round_robin:
    reachable_only: true
  worker: 1
  required_acks: 1
  compression: gzip
  compression_level: 4
  max_message_bytes: 10000000
processors:
  - drop_fields:
     fields:
     - beat
     - host
     - input
     - source
     - offset
     - prospector

启动服务

 

2. logstash配置:

input {
  kafka{
    bootstrap_servers => "log1:9092"
    topics => ["test_kafka"]
    codec => "json"
  }
}

output {
  elasticsearch {
    hosts => "log1:9200"
    index => "test_kafka-%{+YYYY.MM.dd}"
  }
}

启动服务

nohup ./logstash -f es.conf &

 

3.使用kibana验证

可以手动模拟生成日志:

echo "手动echo测试" >> /var/log/messages


 



Viewing all articles
Browse latest Browse all 780

Trending Articles


Vimeo 10.7.1 by Vimeo.com, Inc.


UPDATE SC IDOL: TWO BECOME ONE


KASAMBAHAY BILL IN THE HOUSE


Girasoles para colorear


Presence Quotes – Positive Quotes


EASY COME, EASY GO


Love with Heart Breaking Quotes


Re:Mutton Pies (lleechef)


Ka longiing longsem kaba skhem bad kaba khlain ka pynlong kein ia ka...


Vimeo 10.7.0 by Vimeo.com, Inc.


FORECLOSURE OF REAL ESTATE MORTGAGE


FORTUITOUS EVENT


Pokemon para colorear


Sapos para colorear


Smile Quotes


Letting Go Quotes


Love Song lyrics that marks your Heart


RE: Mutton Pies (frankie241)


Hato lada ym dei namar ka jingpyrshah jong U JJM Nichols Roy (Bah Joy) ngin...


Long Distance Relationship Tagalog Love Quotes