你的位置:首页 > 数据库

[数据库]安装 logstash 2.2.0 和 elasticsearch 2.2.0


本文内容

  • Elasticsearch
  • logstash

本文介绍安装 logstash 2.2.0 和 elasticsearch 2.2.0,操作系统环境版本是 CentOS/Linux 2.6.32-504.23.4.el6.x86_64。

安装 JDK 是必须的,一般操作系统都会有,只是版本的问题,后面会提到。

之所以,Kibana 只是一个 UI,暂不介绍。因为,最公司需要分析所有系统的日志,才搞 ELK,但是人员有限,三个框架都研究,不太现实。

Elasticsearch

Elasticsearch主页

Elasticsearch(简称,ES)提供 ZIP、TAR、DEB 和 RPM 包。但 Github 上提供了一个针对中文环境的 Elasticsearch-RTF,RTF 即 Ready To Fly,它是一个针对中文的发行版,换句话说,帮你入门的。本文针对 Elasticsearch-RTF 为例。基本上,elasticsearch 解压后就能使用。

2016-02-26_150017

假设你已经从 Github 上下载 elasticsearch-rtf,名为 elasticsearch-master.zip,并上传到你的 Linux 服务器 /usr/local/elasticsearch目录(如果没有,就用 mkdir 命令创建一个)。

  • 现在,解压,并重新命名文件夹:
[root@vcyber local]# cd /usr/local/elasticsearch
 
[root@vcyber local]# unzip elasticsearch-master.zip
 
[root@vcyber elasticsearch]# ls
 
elasticsearch-master elasticsearch-master.zip
 
[root@vcyber local]# mv elasticsearch-master elasticsearch
 
[root@vcyber elasticsearch]# ls
 
elasticsearch elasticsearch-master.zip


  • 尝试运行 elasticsearch:

Linux 环境:

[root@vcyber elasticsearch]# pwd
 
/usr/local/elasticsearch/elasticsearch
 
[root@vcyber elasticsearch]# bin/elasticsearch


windows 环境,执行相应的 .bat 文件,即 elasticsearch.bat。

但报错了:

[root@vcyber elasticsearch]# bin/elasticsearch
 
Exception in thread "main" java.lang.RuntimeException: Java version: Oracle Cooration 1.7.0_51 [Java HotSpot(TM) 64-Bit Server VM 24.51-b03] suffers from crical bug https://bugs.openjdk.java.net/browse/JDK-8024830 which can cause dataorruption.
 
Please upgrade the JVM, see http://www.elastic.co/guide/en/elasticsearch/referce/current/_installation.html for current recommendations.
 
If you absolutely cannot upgrade, please add -XX:-UseSuperWord to the JAVA_OPT environment variable.
 
Upgrading is preferred, this workaround will result in degraded performance.
 
    at org.elasticsearch.bootstrap.JVMCheck.check(JVMCheck.java:123)
 
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:283)
 
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:3
 
Refer to the log for complete error details.


大意是:Java 运行时异常,本机版本 JDK 有 bug……让升级 JVM。如果实在不能升级,就向 JAVA_OPT 环境变量添加 -XX:-UseSuperWord 选项。

于是,看一下本机的Java 版本:

[root@vcyber elasticsearch]# java -version
 
java version "1.7.0_51"
 
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
 
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
 
[root@vcyber elasticsearch]# echo $JAVA_HOME
 
/usr/java/jdk1.7.0_51
 
[root@vcyber elasticsearch]#
 


版本是 1.7.0_51。再在官网查了一下,说:“Elasticsearch requires at least Java 7. Specifically as of this writing, it is recommended that you use the Oracle JDK version 1.8.0_72. Java installation varies from platform to platform so we won’t go into those details here. Oracle’s recommended installation documentation can be found on Oracle’s website. Suffice to say, before you install Elasticsearch, please check your Java version first by running (and then install/upgrade accordingly if needed):”,大意是,ES 至少要求 7,推荐使用 1.8.0_72。

  • 那就删除之前的版本,按个新的吧。先删掉之前的 JDK,然后再用 yum 按个新的:
[root@vcyber elasticsearch]# yum list installed | grep java
 
[root@vcyber elasticsearch]# yum list installed | grep jdk
 
jdk.x86_64              2000:1.7.0_51-fcs        installed
 
[root@vcyber elasticsearch]# yum -y remove jdk.x86_64
 
……
 
[root@vcyber elasticsearch]#yum -y install java-1.8.0-openjdk*
 
……


注意:java-1.8.0-openjdk*”,后面有个星号,即安装 java 全部相关的东西~

  • 安装完成后,设置 JDK 的环境变量:
[root@vcyber elasticsearch]# export JAVA_HOME=/usr/lib/jvm/java-1.8.0
 
[root@vcyber elasticsearch]# export PATH=$JAVA_HOME/bin:$PATH
 
[root@vcyber elasticsearch]# export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
 
[root@vcyber elasticsearch]# java -version
 
openjdk version "1.8.0_71"
 
OpenJDK Runtime Environment (build 1.8.0_71-b15)
 
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
 
[root@vcyber elasticsearch]# echo $JAVA_HOME
 
/usr/lib/jvm/java-1.8.0
 
[root@vcyber elasticsearch]#


另外,JDK 安装在了我机器的 /usr/lib/jvm 目录下,自己确认一下你的路径。

  • 再次运行:
[root@vcyber elasticsearch]# bin/elasticsearch
 
Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.
 
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:93)
 
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:144)
 
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
 
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
 
Refer to the log for complete error details.
 
[root@vcyber elasticsearch]#


  • 又报错,elasticsearch 不能用 root 用户运行,那就建立一个:
[root@vcyber elasticsearch]# groupadd es
 
[root@vcyber elasticsearch]# useradd -g es es
 
[root@vcyber elasticsearch]# passwd es
 
Changing password for user es.
 
New password:
 
BAD PASSWORD: it is WAY too short
 
BAD PASSWORD: is too simple
 
Retype new password:
 
passwd: all authentication tokens updated successfully.
 
[root@vcyber elasticsearch]#
 
[root@vcyber elasticsearch]# chown -R root .
 
[root@vcyber elasticsearch]# chown -R es .
 
[root@vcyber elasticsearch]# chgrp -R es .
 
[root@vcyber elasticsearch]# ls -l
 
total 4
 
drwxr-xr-x 7 es es 4096 Mar 1 03:07 elasticsearch
 
[root@vcyber elasticsearch]#


  • 重新打开一个终端,用 es 用户登录,并运行 elasticsearch:
[root@vcyber ~]$ cd /usr/local/elasticsearch/elasticsearch
 
[root@vcyber elasticsearch]$ bin/elasticsearch
 
[2016-03-01 05:11:48,413][WARN ][bootstrap        ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
 
[2016-03-01 05:11:48,750][INFO ][node           ] [Googam] version[2.1.1], pid[15042], build[40e2c53/2015-12-15T13:05:55Z]
 
[2016-03-01 05:11:48,750][INFO ][node           ] [Googam] initializing ...
 
[2016-03-01 05:11:49,088][INFO ][plugins         ] [Googam] loaded [elasticsearch-analysis-ik, elasticsearch-analysis-mmseg, elasticsearch-analysis-stconvert, elasticsearch-analysis-pinyin], sites []
 
[2016-03-01 05:11:49,121][INFO ][env           ] [Googam] using [1] data paths, mounts [[/ (/dev/mapper/vg_vcyber-lv_root)]], net usable_space [26.1gb], net total_space [34.8gb], spins? [possibly], types [ext4]
 
[2016-03-01 05:11:51,119][INFO ][mmseg-analyzer      ] [Dict Loading] chars loaded time=42ms, line=12638, on file=chars.dic
 
……


  • 此时,在另一个终端,访问 elasticsearch:
[root@vcyber elasticsearch]# curl -X GET http://localhost:9200
 
{
 
 "name" : "Captain Savage",
 
 "cluster_name" : "elasticsearch",
 
 "version" : {
 
  "number" : "2.1.1",
 
  "build_hash" : "40e2c53a6b6c2972b3d13846e450e66f4375bd71",
 
  "build_timestamp" : "2015-12-15T13:05:55Z",
 
  "build_snapshot" : false,
 
  "lucene_version" : "5.3.1"
 
 },
 
 "tagline" : "You Know, for Search"
 
}
 
[root@vcyber elasticsearch]#
 


ES 已经安装成功。

但此时的 ES 不能通过IP访问,所以,你要修改 config/elasticsearch.yml。找到“network.host”行,那个示例,添加一行:

network.host: your id address


就能通过IP,或浏览器访问。


logstash

logstash主页

logstash 提供各种安装包,包括 tar.gz,ZIP,DEB 和 RPM。另外,又提供了一个包含所有插件的压缩包——logstash-all-plugins-2.2.0.tar.gz 。本文以它为例。解压后,配置 logstash,运行即可。

2016-02-26_135910

2016-02-26_144944

  • 在 /usr/local 下创建目录 logstash:
[root@vcyber local]# cd /usr/local
 
[root@vcyber local]# pwd
 
/usr/local
 
[root@vcyber local]# mkdir logstash
 
[root@vcyber local]#


  • 上传 logstash 到该目录。然后,解压并配置 logstash 配置文件:
[root@vcyber local]# cd logstash
 
[root@vcyber logstash]# ls
 
logstash-all-plugins-2.2.0.tar.gz
 
[root@vcyber logstash]# tar zxf logstash-all-plugins-2.2.0.tar.gz
 
logstash-2.2.0 logstash-all-plugins-2.2.0.tar.gz
 
[root@vcyber logstash]# cd logstash-2.2.0
 
[root@vcyber logstash-2.2.0]# vi logstash.conf
 
[root@vcyber logstash-2.2.0]#


文件内容如下:

input{
 
  stdin{}
 
}
 
output{
 
  stdout{}
 
}


这个配置是最简单,输入是终端命令行,输出也是终端命令行。

  • 运行 logstash:
[root@vcyber logstash-2.2.0]# bin/logstash agent -f logstash.conf
 
hello world
 
hello world 2
 
Settings: Default pipeline workers: 2
 
Logstash startup completed
 
2016-02-28T22:37:06.130Z vcyber hello world
 
2016-02-28T22:37:06.132Z vcyber hello world 2
 


logst 已经成功安装。其中,agent 表示运行Agent模式,-f 表示指定配置文件,-p 表示端口,命令行参数可以参考logstash command-lines flags。

  • 另外,你可以查看 logstash 都安装了哪些插件。
[root@vcyber logstash-2.2.0]# bin/plugin list
 
logstash-codec-avro
 
logstash-codec-cef
 
logstash-codec-cloudfront
 
logstash-codec-cloudtrail
 
logstash-codec-collectd
 
logstash-codec-compress_spooler
 
logstash-codec-dots
 
logstash-codec-edn
 
logstash-codec-edn_lines
 
logstash-codec-es_bulk
 
logstash-codec-fluent
 
logstash-codec-graphite
 
logstash-codec-gzip_lines
 
logstash-codec-json
 
logstash-codec-json_lines
 
logstash-codec-line
 
logstash-codec-msgpack
 
logstash-codec-multiline
 
logstash-codec-netflow
 
logstash-codec-nmap
 
logstash-codec-oldlogstashjson
 
logstash-codec-plain
 
logstash-codec-rubydebug
 
logstash-codec-s3plain
 
logstash-codec-spool
 
logstash-filter-aggregate
 
logstash-filter-alter
 
logstash-filter-anonymize
 
logstash-filter-checksum
 
logstash-filter-cidr
 
logstash-filter-cipher
 
logstash-filter-clone
 
logstash-filter-collate
 
logstash-filter-csv
 
logstash-filter-date
 
logstash-filter-de_dot
 
logstash-filter-dns
 
logstash-filter-drop
 
logstash-filter-elapsed
 
logstash-filter-elasticsearch
 
logstash-filter-environment
 
logstash-filter-extractnumbers
 
logstash-filter-fingerprint
 
logstash-filter-geoip
 
logstash-filter-grok
 
logstash-filter-i18n
 
logstash-filter-json
 
logstash-filter-json_encode
 
logstash-filter-kv
 
logstash-filter-metaevent
 
logstash-filter-metricize
 
logstash-filter-metrics
 
logstash-filter-multiline
 
logstash-filter-mutate
 
logstash-filter-oui
 
logstash-filter-prune
 
logstash-filter-punct
 
logstash-filter-range
 
logstash-filter-ruby
 
logstash-filter-sleep
 
logstash-filter-split
 
logstash-filter-syslog_pri
 
logstash-filter-throttle
 
logstash-filter-tld
 
logstash-filter-translate
 
logstash-filter-unique
 
logstash-filter-urldecode
 
logstash-filter-useragent
 
logstash-filter-uuid
 
logstash-filter-  
 
logstash-filter-zeromq
 
logstash-input-beats
 
logstash-input-cloudwatch
 
logstash-input-couchdb_changes
 
logstash-input-elasticsearch
 
logstash-input-eventlog
 
logstash-input-exec
 
logstash-input-file
 
logstash-input-fluentd
 
logstash-input-ganglia
 
logstash-input-gelf
 
logstash-input-gemfire
 
logstash-input-generator
 
logstash-input-github
 
logstash-input-graphite
 
logstash-input-heartbeat
 
logstash-input-http
 
logstash-input-http_poller
 
logstash-input-imap
 
logstash-input-irc
 
logstash-input-jdbc
 
logstash-input-jmx
 
logstash-input-kafka
 
logstash-input-log4j
 
logstash-input-lumberjack
 
logstash-input-meetup
 
logstash-input-pipe
 
logstash-input-puppet_facter
 
logstash-input-rabbitmq
 
logstash-input-redis
 
logstash-input-relp
 
logstash-input-rss
 
logstash-input-s3
 
logstash-input-salesforce
 
logstash-input-snmptrap
 
logstash-input-sqlite
 
logstash-input-sqs
 
logstash-input-stdin
 
logstash-input-stomp
 
logstash-input-syslog
 
logstash-input-tcp
 
logstash-input-twitter
 
logstash-input-udp
 
logstash-input-unix
 
logstash-input-varnishlog
 
logstash-input-websocket
 
logstash-input-wmi
 
logstash-input-xmpp
 
logstash-input-zenoss
 
logstash-input-zeromq
 
logstash-output-boundary
 
logstash-output-circonus
 
logstash-output-cloudwatch
 
logstash-output-csv
 
logstash-output-datadog
 
logstash-output-datadog_metrics
 
logstash-output-elasticsearch
 
logstash-output-elasticsearch-ec2
 
logstash-output-elasticsearch_http
 
logstash-output-elasticsearch_java
 
logstash-output-email
 
logstash-output-exec
 
logstash-output-file
 
logstash-output-ganglia
 
logstash-output-gelf
 
logstash-output-google_bigquery
 
logstash-output-google_cloud_storage
 
logstash-output-graphite
 
logstash-output-graphtastic
 
logstash-output-hipchat
 
logstash-output-http
 
logstash-output-influxdb
 
logstash-output-irc
 
logstash-output-juggernaut
 
logstash-output-kafka
 
logstash-output-librato
 
logstash-output-loggly
 
logstash-output-lumberjack
 
logstash-output-metriccatcher
 
logstash-output-mongodb
 
logstash-output-nagios
 
logstash-output-nagios_nsca
 
logstash-output-null
 
logstash-output-opentsdb
 
logstash-output-pagerduty
 
logstash-output-pipe
 
logstash-output-rabbitmq
 
logstash-output-redis
 
logstash-output-redmine
 
logstash-output-riemann
 
logstash-output-s3
 
logstash-output-sns
 
logstash-output-solr_http
 
logstash-output-sqs
 
logstash-output-statsd
 
logstash-output-stdout
 
logstash-output-stomp
 
logstash-output-syslog
 
logstash-output-tcp
 
logstash-output-udp
 
logstash-output-websocket
 
logstash-output-xmpp
 
logstash-output-zabbix
 
logstash-output-zeromq
 
logstash-patterns-core
 
[root@vcyber logstash-2.2.0]#