ELK-日志分析系统

简述:

ELk由Elasticsearch、Logstash和Kibana三部分组件组成;

Elasticsearch: 存储各类日志;
Logstash: logstash server端用来搜集日志;
Kibana: web化接口用作查寻和可视化日志;
官网下载>>

环境准备:

三台 Centos6.5 的机器(模拟分布式集群):
Linux-node-1: 192.168.31.63
Linux-node-2: 192.168.31.78
Linux-node-3: 192.168.31.236

Java-1.8.0
Elasticsearch 5.5.0

安装:

[root@Linux-node-1 ~]# yum -y install java-1.8.0
[root@Linux-node-1 ~]# java -version
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-b16)
OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)
[root@Linux-node-1 ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.tar.gz
[root@Linux-node-1 ~]# tar zxf elasticsearch-5.5.0.tar.gz -C /usr/local/
[root@Linux-node-1 ~]# mv /usr/local/elasticsearch-5.5.0 /usr/local/elasticsearch

因为2.x版本开始基于安全考虑,elasticsearch无法使用root启动,需要创建新的用户

[root@Linux-node-1 ~]# useradd elk
[root@Linux-node-1 ~]# mkdir -p /usr/local/elasticsearch/{data,logs}
[root@Linux-node-1 ~]# chown elk:elk -R /usr/local/elasticsearch/
[root@Linux-node-1 ~]# vim /usr/local/elasticsearch/config/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: My-ELK     //集群名称,根据此名称一致的添加到集群里
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: Linux-node-1           //集群里的机器名称
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /usr/local/elasticsearch/data    // 数据存储路径(需要创建)
#
# Path to log files:
#
path.logs: /usr/local/elasticsearch/logs     //日志存储路径(需要创建)
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#因为Centos6不支持SecComp,而ES5.2.1默认bootstrap.system_call_filter为true进行检测,所以导致检测失败,失败后直接导致ES不能启动
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["0.0.0.0"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
discovery.zen.ping.unicast.hosts: ["127.0.0.1"]
#
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
[root@Linux-node-1 ~]# 
[root@Linux-node-1 ~]# egrep -v '^$|^#' /usr/local/elasticsearch/config/elasticsearch.yml 
cluster.name: My-ELK
node.name: Linux-node-1
path.data: /usr/local/elasticsearch/data
path.logs: /usr/local/elasticsearch/logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["127.0.0.1"]
[root@Linux-node-1 ~]# 

接下来我们启动看看:

启动报错:

[root@Linux-node-1 ~]# su elk
[elk@Linux-node-1 root]$ /usr/local/elasticsearch/bin/elasticsearch
...  ...
ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
max number of threads [1024] for user [hdfs] is too low, increase to at least [2048]
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
system call filters failed to install; check the logs and fix your configuration or disable system call filters

起不来了吧,根据以上报错信息我们还得调下:

第一个问题:

max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

原因:无法创建本地文件问题,用户最大可创建文件数太小

[root@h001 elasticsearch]# vim /etc/security/limits.conf
 添加如下内容:
 * soft nofile 65536
 * hard nofile 131072
 * soft nproc 2048
 * hard nproc 4096

第二个问题:

max number of threads [1024] for user [hdfs] is too low, increase to at least [2048]

原因:无法创建本地线程问题,用户最大可创建线程数太小

[root@h001 elasticsearch]# vim /etc/security/limits.d/90-nproc.conf
修改如下内容:
* soft nproc 1024
#修改为
* soft nproc 2048  

第三个问题:

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
原因:最大虚拟内存太小

[root@h001 elasticsearch]# vim /etc/sysctl.conf 
添加下面配置:
vm.max_map_count=655360
并执行命令:
sysctl -p

再启动:

[root@Linux-node-1 ~]# su elk
[elk@Linux-node-1 root]$ /usr/local/elasticsearch/bin/elasticsearch

验证:

[root@Linux-node-1 ~]# curl http://192.168.31.63:9200
{
  "name" : "Linux-node-1",
  "cluster_name" : "My-ELK",
  "cluster_uuid" : "eiGtxlQfQY2m1rYAZzNd-A",
  "version" : {
    "number" : "5.5.0",
    "build_hash" : "260387d",
    "build_date" : "2017-06-30T23:16:05.735Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}
[root@Linux-node-1 ~]# 
comments powered by Disqus