基于Centos7系統(tǒng)一鍵部署EFK服務(wù)，值得收藏
作者：波波說運(yùn)維
最近平臺(tái)EFK版本均作了升級(jí)，平臺(tái)采用EFK(ElasticSearch-7.6.2 + FileBeat-7.6.2 + Kibana-7.6.2)架構(gòu)。這里建議三個(gè)組件主次版本保持一致。考慮到服務(wù)器比較多，所以寫成腳本來批量部署。
最近平臺(tái)EFK版本均作了升級(jí)，平臺(tái)采用EFK(ElasticSearch-7.6.2 + FileBeat-7.6.2 + Kibana-7.6.2)架構(gòu)。這里建議三個(gè)組件主次版本保持一致。考慮到服務(wù)器比較多，所以寫成腳本來批量部署。
腳本內(nèi)容大家看一下function就行了..
架構(gòu)
EFK采用集中式的日志管理架構(gòu)
- elasticsearch：一個(gè)開源分布式搜索引擎，提供搜集、分析、存儲(chǔ)數(shù)據(jù)三大功能。它的特點(diǎn)有：分布式，零配置，自動(dòng)發(fā)現(xiàn)，索引自動(dòng)分片，索引副本機(jī)制，restful風(fēng)格接口，多數(shù)據(jù)源，自動(dòng)搜索負(fù)載等。
- kibana：可以為L(zhǎng)ogstash 、Beats和ElasticSearch提供友好的日志分析Web 界面，可以幫助匯總、分析和搜索重要數(shù)據(jù)日志。
- filebeat：輕量級(jí)日志采集器。需要在每個(gè)應(yīng)用服務(wù)器配置filebeat，來采集日志，并輸出到elasticsearch
一鍵部署KIBANA腳本
- function install_es7_el7()
- {
- echo ""
- echo -e "\033[33m****************************************************安裝ElasticSearch 7.6.2*****************************************************\033[0m"
- #action "********初始化JAVA環(huán)境********" /bin/true
- #install_jdk
- #下載包
- if [ -f /opt/elasticsearch-7.6.2-x86_64.rpm ] && [ -f /opt/elasticsearch-analysis-ik-7.6.2.zip ] ;then
- echo "*****存在ElasticSearch 7.6.2安裝包，無需下載*****"
- else
- ping -c 4 artifacts.elastic.co >/dev/null 2>&1
- if [ $? -eq 0 ];then
- wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.2-x86_64.rpm -O /opt/elasticsearch-7.6.2-x86_64.rpm
- wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.6.2/elasticsearch-analysis-ik-7.6.2.zip -O /opt/elasticsearch-analysis-ik-7.6.2.zip
- else
- echo "please download ES7 package manual !"
- exit $?
- fi
- fi
- #安裝es7.6
- action "********安裝ElasticSearch 7.6.2服務(wù)********" /bin/true
- chmod u+x /opt/elasticsearch-7.6.2-x86_64.rpm && rpm -ivh /opt/elasticsearch-7.6.2-x86_64.rpm
- #建目錄及授權(quán)
- mkdir -p $ES_HOME/data && mkdir -p $ES_HOME/log
- chown -R elasticsearch:elasticsearch $ES_HOME && chmod -R 755 $ES_HOME
- #修改ES配置文件
- cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml_bak &>/dev/null
- cat > /etc/elasticsearch/elasticsearch.yml << EOF
- cluster.name: es-cluster
- # 設(shè)定本機(jī)節(jié)點(diǎn)名
- node.name: es_node
- # 設(shè)定集群主節(jié)點(diǎn)清單
- cluster.initial_master_nodes: ["es_node"]
- path.data: ${ES_HOME}/data
- path.logs: ${ES_HOME}/log
- bootstrap.memory_lock: false
- bootstrap.system_call_filter: false
- # 允許從其它機(jī)器訪問
- network.host: 0.0.0.0
- http.port: ${ES_PORT}
- discovery.zen.ping.unicast.hosts: ["${ES_IP}:${ES_PORT}"]
- EOF
- #安裝分詞器：ik-analyzer插件
- #默認(rèn)情況下，ES使用內(nèi)置的標(biāo)準(zhǔn)分詞器，對(duì)文本進(jìn)行解析。但是對(duì)于中文，其會(huì)拆解為一個(gè)一個(gè)的漢字，最終失去了分詞的意義，所以安裝分詞器：ik-analyzer插件
- action "********安裝ik-analyzer插件********" /bin/true
- mkdir -p /usr/share/elasticsearch/plugins/ik
- unzip /opt/elasticsearch-analysis-ik-7.6.2.zip -d /usr/share/elasticsearch/plugins/ik/ &>/dev/null
- chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/plugins/ && chmod -R 755 /usr/share/elasticsearch/plugins/
- sleep 2
- #在filebeat配置文件中為索引模板添加默認(rèn)分詞器屬性。未來新創(chuàng)建的索引均引用此屬性:
- #setup.template.settings:
- # index.analysis.analyzer.default.type: "ik_max_word"
- # index.analysis.analyzer.default_search.type: "ik_max_word"
- #setup.template.overwrite: true
- #在Linux終端中執(zhí)行如下命令，為現(xiàn)有所有索引，設(shè)置默認(rèn)分詞器屬性:
- #curl -X POST "172.16.20.143:9200/_all/_close"
- #curl -X PUT -H'Content-Type: application/json' 'http://172.16.20.143:9200/_all/_settings?preserve_existing=true' -d '{
- # "index.analysis.analyzer.default.type" : "ik_max_word",
- # "index.analysis.analyzer.default_search.type" : "ik_max_word"
- #}'
- #curl -X POST "172.16.20.143:9200/_all/_open"
- #啟動(dòng)ES并初始化數(shù)據(jù)
- action "********啟動(dòng)es并初始化數(shù)據(jù)********" /bin/true
- systemctl daemon-reload && systemctl enable elasticsearch.service
- systemctl restart elasticsearch.service
- es_version=`curl -XGET ${ES_IP}:${ES_PORT}`
- echo -e "\033[33m**************************************************完成ElasticSearch 7.6.2安裝***************************************************\033[0m"
- cat > /tmp/es7.log << EOF
- ES服務(wù)器IP： ${ES_IP}
- ES服務(wù)器端口:${ES_PORT}
- ES數(shù)據(jù)目錄： ${ES_HOME}/data
- ES日志目錄： ${ES_HOME}/log
- ES詳細(xì)信息： ${es_version}
- EOF
- cat /tmp/es7.log
- echo -e "\e[1;31m 以上信息10秒后消失，保存在/tmp/es7.log文件下 \e[0m"
- echo -e "\033[33m************************************************************************************************************************\033[0m"
- echo ""
- sleep 10
- }
一鍵部署Elasticsearch腳本
- function install_es7_el7()
- {
- echo ""
- echo -e "\033[33m****************************************************安裝ElasticSearch 7.6.2*****************************************************\033[0m"
- #action "********初始化JAVA環(huán)境********" /bin/true
- #install_jdk
- #下載包
- if [ -f /opt/elasticsearch-7.6.2-x86_64.rpm ] && [ -f /opt/elasticsearch-analysis-ik-7.6.2.zip ] ;then
- echo "*****存在ElasticSearch 7.6.2安裝包，無需下載*****"
- else
- ping -c 4 artifacts.elastic.co >/dev/null 2>&1
- if [ $? -eq 0 ];then
- wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.2-x86_64.rpm -O /opt/elasticsearch-7.6.2-x86_64.rpm
- wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.6.2/elasticsearch-analysis-ik-7.6.2.zip -O /opt/elasticsearch-analysis-ik-7.6.2.zip
- else
- echo "please download ES7 package manual !"
- exit $?
- fi
- fi
- #安裝es7.6
- action "********安裝ElasticSearch 7.6.2服務(wù)********" /bin/true
- chmod u+x /opt/elasticsearch-7.6.2-x86_64.rpm && rpm -ivh /opt/elasticsearch-7.6.2-x86_64.rpm
- #建目錄及授權(quán)
- mkdir -p $ES_HOME/data && mkdir -p $ES_HOME/log
- chown -R elasticsearch:elasticsearch $ES_HOME && chmod -R 755 $ES_HOME
- #修改ES配置文件
- cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml_bak &>/dev/null
- cat > /etc/elasticsearch/elasticsearch.yml << EOF
- cluster.name: es-cluster
- # 設(shè)定本機(jī)節(jié)點(diǎn)名
- node.name: es_node
- # 設(shè)定集群主節(jié)點(diǎn)清單
- cluster.initial_master_nodes: ["es_node"]
- path.data: ${ES_HOME}/data
- path.logs: ${ES_HOME}/log
- bootstrap.memory_lock: false
- bootstrap.system_call_filter: false
- # 允許從其它機(jī)器訪問
- network.host: 0.0.0.0
- http.port: ${ES_PORT}
- discovery.zen.ping.unicast.hosts: ["${ES_IP}:${ES_PORT}"]
- EOF
- #安裝分詞器：ik-analyzer插件
- #默認(rèn)情況下，ES使用內(nèi)置的標(biāo)準(zhǔn)分詞器，對(duì)文本進(jìn)行解析。但是對(duì)于中文，其會(huì)拆解為一個(gè)一個(gè)的漢字，最終失去了分詞的意義，所以安裝分詞器：ik-analyzer插件
- action "********安裝ik-analyzer插件********" /bin/true
- mkdir -p /usr/share/elasticsearch/plugins/ik
- unzip /opt/elasticsearch-analysis-ik-7.6.2.zip -d /usr/share/elasticsearch/plugins/ik/ &>/dev/null
- chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/plugins/ && chmod -R 755 /usr/share/elasticsearch/plugins/
- sleep 2
- #在filebeat配置文件中為索引模板添加默認(rèn)分詞器屬性。未來新創(chuàng)建的索引均引用此屬性:
- #setup.template.settings:
- # index.analysis.analyzer.default.type: "ik_max_word"
- # index.analysis.analyzer.default_search.type: "ik_max_word"
- #setup.template.overwrite: true
- #在Linux終端中執(zhí)行如下命令，為現(xiàn)有所有索引，設(shè)置默認(rèn)分詞器屬性:
- #curl -X POST "172.16.20.143:9200/_all/_close"
- #curl -X PUT -H'Content-Type: application/json' 'http://172.16.20.143:9200/_all/_settings?preserve_existing=true' -d '{
- # "index.analysis.analyzer.default.type" : "ik_max_word",
- # "index.analysis.analyzer.default_search.type" : "ik_max_word"
- #}'
- #curl -X POST "172.16.20.143:9200/_all/_open"
- #啟動(dòng)ES并初始化數(shù)據(jù)
- action "********啟動(dòng)es并初始化數(shù)據(jù)********" /bin/true
- systemctl daemon-reload && systemctl enable elasticsearch.service
- systemctl restart elasticsearch.service
- es_version=`curl -XGET ${ES_IP}:${ES_PORT}`
- echo -e "\033[33m**************************************************完成ElasticSearch 7.6.2安裝***************************************************\033[0m"
- cat > /tmp/es7.log << EOF
- ES服務(wù)器IP： ${ES_IP}
- ES服務(wù)器端口:${ES_PORT}
- ES數(shù)據(jù)目錄： ${ES_HOME}/data
- ES日志目錄： ${ES_HOME}/log
- ES詳細(xì)信息： ${es_version}
- EOF
- cat /tmp/es7.log
- echo -e "\e[1;31m 以上信息10秒后消失，保存在/tmp/es7.log文件下 \e[0m"
- echo -e "\033[33m************************************************************************************************************************\033[0m"
- echo ""
- sleep 10
- }
一鍵部署filebeat腳本
- function install_filebeat7_el7()
- {
- echo ""
- echo -e "\033[33m****************************************************安裝Filebeat 7.6.2*****************************************************\033[0m"
- #下載包
- if [ -f /opt/filebeat-7.6.2-x86_64.rpm ] ;then
- echo "*****存在Filebeat 7.6.2安裝包，無需下載*****"
- else
- ping -c 4 artifacts.elastic.co >/dev/null 2>&1
- if [ $? -eq 0 ];then
- wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.6.2-x86_64.rpm -O /opt/filebeat-7.6.2-x86_64.rpm
- else
- echo "please download Filebeat7.6 package manual !"
- exit $?
- fi
- fi
- #安裝filebeat7.6
- action "********安裝filebeat 7.6.2服務(wù)********" /bin/true
- chmod u+x /opt/filebeat-7.6.2-x86_64.rpm && rpm -ivh /opt/filebeat-7.6.2-x86_64.rpm
- #修改kibana配置文件
- cp /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml_bak
- cat > /etc/filebeat/filebeat.yml << EOF
- filebeat.inputs:
- - type: log
- enabled: true
- paths:
- - /lcp_logs/*.log
- filebeat.config.modules:
- path: /etc/filebeat/modules.d/*.yml
- reload.enabled: false
- setup.template.settings:
- #number_of_shards 是數(shù)據(jù)分片數(shù)，默認(rèn)為5，有時(shí)候設(shè)置為3
- index.number_of_shards: 3
- index.analysis.analyzer.default.type: "ik_max_word"
- index.analysis.analyzer.default_search.type: "ik_max_word"
- setup.template.overwrite: true
- setup.kibana:
- host: "${KIBANA_IP}:${KIBANA_PORT}"
- output.elasticsearch:
- hosts: ["${ES_IP}:${ES_PORT}"]
- ilm.enabled: true
- ilm.rollover_alias: "fsl_uat.prod1"
- ilm.pattern: "{now/d}-000001"
- processors:
- - add_host_metadata: ~
- - add_cloud_metadata: ~
- EOF
- #啟動(dòng)filebeat并初始化數(shù)據(jù)
- action "********啟動(dòng)filebeat并初始化數(shù)據(jù)********" /bin/true
- systemctl daemon-reload && systemctl enable filebeat.service
- systemctl restart filebeat.service
- #nohup ./filebeat -e -c filebeat.yml >/dev/null 2>&1 &
- echo -e "\033[33m**************************************************完成Filebeat 7.6.2安裝***************************************************\033[0m"
- cat > /tmp/filebeat7.log << EOF
- filebeat版本： 7.6.2
- filebeat路徑: /usr/bin/filebeat
- filebeat配置kibana： ${KIBANA_IP}:${KIBANA_PORT}
- filebeat配置elasticsearch： ${ES_IP}:${ES_PORT}
- EOF
- cat /tmp/filebeat7.log
- echo -e "\e[1;31m 以上信息10秒后消失，保存在/tmp/filebeat7.log文件下 \e[0m"
- echo -e "\033[33m************************************************************************************************************************\033[0m"
- echo ""
- sleep 10
- }
責(zé)任編輯：武曉燕
來源：
今日頭條
