单节点elk+grafana搭建和采集nginx访问日志

需求：展现nginx打印的信息（如 pv uv http状态码 url前十）

此博客架构：es+kibana 在一台，logstash+nginx 在另外一台
java

1、node

安装包提供：linux

elasticsearch-7.2.0-linux-x86_64.tar.gz
logstash-7.2.0.tar.gz
kibana-7.2.0-linux-x86_64.tar.gz
grafana-7.2.0-1.x86_64.rpm
nginx

2、部署es前提条件json

1. echo "vm.max_map_count=655360" >>/etc/sysctl.conf
2. sysctl -p
3. vim /etc/security/limits.conf
4.    * soft core unlimited
5.    * hard core unlimited
6.    * soft nofile 1048576
7.    * hard nofile 1048576
8.    * soft nproc 65536
9.    * hard nproc 65536
10.    * soft sigpending 255983
11.    * hard sigpending 255983
12.    * soft memlock unlimited
13.    * hard memlock unlimited
14. vim /etc/security/limits.d/20-nproc.conf
15. * soft nproc 65536

16. * hard nproc 65536bootstrap

    3、部署esvim

17. useradd elastic #不能用root起服务，若是root起须要修改相关配置centos

18. 因为咱们才用的是elk7.2 因此jdk要大于java8,本环境安装java11ruby

19. yum install java-11-openjdk -y微信

20. tar -zxvf elasticsearch-7.2.0-linux-x86_64.tar.gz
    mv elasticsearch-7.2.0 es
    mv elasticsearch.yml elasticsearch.yml.bak
    vim elasticsearch.yml
    [root@localhost config]# grep -vE '^#|^$'  elasticsearch.yml
    cluster.name: my-application
    node.name: node-1
    path.data: /data/es/data
    path.logs: /data/es/logs
    bootstrap.memory_lock: false
    bootstrap.system_call_filter: false
    network.host: 0.0.0.0
    cluster.initial_master_nodes: ["node-1"]
    
    chown -R elastic.elastic /data/es
    
    进入elastic启动服务：

21. nohup ./bin/elasticsearch -d &

测试：curl -XGET 'localhost:9200/?pretty'

 查看index : curl 'localhost:9200/_cat/indices?v'

4、部署logstash

tar -zxvf logstash-7.2.0.tar.gz

测试： 输入到控制台  ../bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
      输入到es     ../bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["localhost:9200"]} }'

5、收集nginx日志

cat /etc/yum.repos.d/nginx.repo
#在文件中写入如下内容：
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

yum install nginx -y

日志格式：

  log_format main   '{"@timestamp":"$time_iso8601",'
                        '"@source":"$server_addr",'
                        '"hostname":"$hostname",'
                        '"ip":"$http_x_forwarded_for",'
                        '"client":"$remote_addr",'
                        '"request_method":"$request_method",'
                        '"scheme":"$scheme",'
                        '"domain":"$server_name",'
                        '"referer":"$http_referer",'
                        '"request":"$request_uri",'
                        '"args":"$args",'
                        '"size":$body_bytes_sent,'
                        '"status": $status,'
                        '"responsetime":$request_time,'
                        '"upstreamtime":"$upstream_response_time",'
                        '"upstreamaddr":"$upstream_addr",'
                        '"http_user_agent":"$http_user_agent",'
                        '"https":"$https",'
                        '"message":"$remote_addr $server_name $status $http_user_agent"'
                        '}';
                        
                        access_log  logs/access.log  main；   #改为上面添加的名称保存后重载Nginxnginx -s reload

配置收集nginx访问日志conf 添加收集nginx的配置vim testng.conf
input {
    file {
        path => [ "/data/log/nginx/*.access.log" ]
        ignore_older => 0
        type => "nginx-log"
        codec => json
    }
}

filter {
    mutate {
      convert => [ "status","integer" ]
      convert => [ "size","integer" ]
      convert => [ "upstreatime","float" ]
      remove_field => "message"
    }
    geoip {
        source => "ip"
    }

}
output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "nginx-log-%{+YYYY.MM.dd}"

    }
stdout{
         codec => rubydebug  #控制台打印日志
       }

}
}

能够加上--configtest参数，测试下配置文件是否有语法错误或配置不当的地方/bin/logstash -f file.conf --configtest
放入后台启动 nohup ../bin/logstash -f testng.conf & 

curl localhost #这个时候应该控制台就有日志打印出来了，再去看es索引 应该就有了

6、部署grafana

yum install grafana-7.2.0-1.x86_64.rpm
#安装插件
grafana-cli plugins install grafana-piechart-panel
grafana-cli plugins install grafana-worldmap-panel
etc/init.d/grafana-server start

导入模板 
 #能够才用这个，不过要对应的改规则
适用于本博客的模板：

7、部署kibana

tar -zxvf kibana-7.2.0-linux-x86_64.tar.gz
cd kibana-7.2.0-linux-x86_64
cd config/
cp kibana.yml kibana.yml.bak
vim kibana.yml
nohup ./bin/kibana & #用非root用户启动

8、展现图(模板右上角能够直接跳转到kibana，要在json里面修改正确的地址)