ELK docker版+Filebeat

之前试过源码和 rpm 安装,但准备及配置相对复杂,不易于快速部署,于是试下 docker 版的,本次用的 ELK 是新版的 7.0.1 ,部署简单但我想实现的功能确折腾了我好几天(网上一堆都是5.X版本的,7.X已经有变化了,以前的方法我试了没成功,...还是自己折腾最靠谱

参考文档>>>

  • 想要的功能 :
  • 支持自定义多索引

准备环境:

  • Centos 7.5 一台
  • IP: 192.168.86.190
  • 关闭 SELINUX 和 Firewalld

[root@localhost ~]# curl http://192.168.86.190:9200
{
  "name" : "elk",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "pACRY6d4QTWXfCLZ7B5OQA",
  "version" : {
    "number" : "7.0.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "e4efcb5",
    "build_date" : "2019-04-29T12:56:03.145736Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.7.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
[root@localhost ~]# 

安装 Filebeat

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.0.1-x86_64.rpm
[root@localhost ~]# rpm -ivh filebeat-7.0.1-x86_64.rpm

[root@localhost ~]# cd /etc/filebeat/
[root@localhost filebeat]# mv filebeat.yml filebeat.yml_bak

[root@localhost filebeat]# vim filebeat.yml 
filebeat.inputs:
- type: log
enabled: true
paths:
	- /var/log/room.log
tags: ["room"]
fields:
	service: room


- type: log
enabled: true
paths:
	- /var/log/1.log
tags: ["test"]
fields:
	service: test

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
output.logstash:
  hosts: ["localhost:5044"]

[root@localhost filebeat]# 
[root@localhost filebeat]#  systemctl start filebeat
[root@localhost filebeat]#  systemctl status filebeat

Filebeat 配置参数参考文献 :
Filebeat 配置介绍
Filebeat 模块与配置

安装 ELK

1.安装docker

 yum -y install docker

2.启动 docker

 systemctl start docker
 systemctl enable docker 

3.拉取 ELK docker 镜像

 docker pull sebp/elk

4.启动 ELK 映射端口

 docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk -d sebp/elk

5.进入 ELK docker 环境,并修改 logstash 配置

[root@localhost ~]# docker exec -it elk /bin/bash
root@486021afab57:/# cd /etc/logstash/conf.d/
root@486021afab57:/etc/logstash/conf.d# ls
02-beats-input.conf  10-syslog.conf  11-nginx.conf  30-output.conf
root@486021afab57:/etc/logstash/conf.d#
logstash input 配置
root@486021afab57:/etc/logstash/conf.d# cat 02-beats-input.conf 
#input {
#  beats {
#    port => 5044
#    ssl => true
#    ssl_certificate => "/etc/pki/tls/certs/logstash-beats.crt"
#    ssl_key => "/etc/pki/tls/private/logstash-beats.key"
#  }
#}

input {
    beats {
      port => 5044
	}
}
root@486021afab57:/etc/logstash/conf.d# 
logstash output 配置
root@486021afab57:/etc/logstash/conf.d# cat 30-output.conf 
#output {
#  elasticsearch {
#    hosts => ["localhost"]
#    manage_template => false
#    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
#  }
#}

output {
if [fields][service] == "room"{
		elasticsearch {
				hosts => ["localhost"]
				index => "room-%{+YYYY.MM.dd}"
		}
}

if [fields][service] == "test"{
		elasticsearch {
				hosts => ["localhost"]
				index => "test-%{+YYYY.MM.dd}"
		}
}

}
root@486021afab57:/etc/logstash/conf.d#
root@486021afab57:/etc/logstash/conf.d# service logstash restart

以上我们部署就完成了,接下来我们验证下刚刚的配置
往 filebeat.xml 定义的 日志 写点东西
[root@localhost ~]# echo "hahahhehexkkfjk" > /var/log/room.log
[root@localhost ~]# echo "1233211234567" > /var/log/1.log
写完看下 filebeat 的状态
[root@localhost ~]# /etc/init.d/filebeat status(执行多几次看看)

[root@localhost filebeat]# /etc/init.d/filebeat status
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
Active: active (running) since 二 2019-06-04 10:48:33 CST; 3s ago
	Docs: https://www.elastic.co/products/beats/filebeat
Main PID: 24256 (filebeat)
Memory: 7.5M
CGroup: /system.slice/filebeat.service
		└─24256 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

6月 04 10:48:33 localhost.localdomain filebeat[24256]: 2019-06-04T10:48:33.122+0800        WARN        beater/filebeat.go:357        Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasti...
6月 04 10:48:33 localhost.localdomain filebeat[24256]: 2019-06-04T10:48:33.122+0800        INFO        crawler/crawler.go:72        Loading Inputs: 2
6月 04 10:48:33 localhost.localdomain filebeat[24256]: 2019-06-04T10:48:33.123+0800        INFO        [monitoring]        log/log.go:117        Starting metrics logging every 30s
6月 04 10:48:33 localhost.localdomain filebeat[24256]: 2019-06-04T10:48:33.124+0800        INFO        log/input.go:138        Configured paths: [/var/log/room.log]
6月 04 10:48:33 localhost.localdomain filebeat[24256]: 2019-06-04T10:48:33.124+0800        INFO        input/input.go:114        Starting input of type: log; ID: 2929081636284132375
6月 04 10:48:33 localhost.localdomain filebeat[24256]: 2019-06-04T10:48:33.125+0800        INFO        log/input.go:138        Configured paths: [/var/log/1.log]
6月 04 10:48:33 localhost.localdomain filebeat[24256]: 2019-06-04T10:48:33.125+0800        INFO        input/input.go:114        Starting input of type: log; ID: 2152830911250164047
6月 04 10:48:33 localhost.localdomain filebeat[24256]: 2019-06-04T10:48:33.126+0800        INFO        crawler/crawler.go:106        Loading and starting Inputs completed. Enabled inputs: 2
6月 04 10:48:33 localhost.localdomain filebeat[24256]: 2019-06-04T10:48:33.126+0800        INFO        cfgfile/reload.go:150        Config reloader started
6月 04 10:48:33 localhost.localdomain filebeat[24256]: 2019-06-04T10:48:33.126+0800        INFO        cfgfile/reload.go:205        Loading of config files completed.
Hint: Some lines were ellipsized, use -l to show in full.
[root@localhost filebeat]# ll /var/log/filebeat/

看到上面有 input 到 logstash 了 那么我看下是否有生成索引了

[root@localhost ~]# curl -XGET 'http://192.168.86.190:9200/_cat/indices'
yellow open room-2019.06.06      mByz6vMVRoqKMGQlA5XPtg 1 1 2011 0 979.9kb 979.9kb
yellow open test-2019.06.06      isVsxx5aTd-6_GHqX_kfBw 1 1   78 0  34.2kb  34.2kb
green  open .kibana_task_manager S2_VqZekRTeyotM7JnkpPQ 1 0    2 0  29.7kb  29.7kb
green  open .kibana_1            3M8dxwvzRWqO3f3_tBtx0g 1 0    5 1  41.3kb  41.3kb
[root@localhost ~]# 

yellow open 看 已经生成索引了,接着我们打开 Kibanna 瞅瞅
浏览器打开 192.168.86.190:5601
index
index_2

ELK 安全加固

Kibana

Kibana 玩法
Kibana 官方文档

comments powered by Disqus