ELK

장점

사전 준비

nginx 설치(샘플용)

sudo yum install nginx -y
sudo service nginx start
curl -i http://localhost
sudo chown -R ec2-user:ec2-user /var/log/nginx /usr/share/nginx/html
echo "<h1>Hello World</h1>" > /usr/share/nginx/html/hello.html

jdk 1.8

sudo yum remove java-1.7.0-openjdk.x86_64 -y
sudo yum install java-1.8.0-openjdk-devel.x86_64 -y

system env

set

sudo vi /etc/security/limits.conf

*

ec2-user hard nofile 65536
ec2-user soft nofile 65536
ec2-user hard nproc 65536
ec2-user soft nproc 65536
sudo vi /etc/rc.local

*

echo 1048575 > /proc/sys/vm/max_map_count
sudo sysctl -w fs.file-max=65536
cat /proc/sys/fs/file-max
sudo reboot

AWS 포트 설정

설치

Elasticsearch 설치

mkdir ~/local
cd ~/local
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.1.tar.gz
tar xvfz elasticsearch-5.6.1.tar.gz
ln -s elasticsearch-5.6.1 elasticsearch
cd elasticsearch
bin/elasticsearch -d
  # 데몬(백그라운드)로 실행. 옵션 -d를 빼면 터미널 접속해 있는 동안만 실행

Kibana 설치

cd ~/local
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.6.1-linux-x86_64.tar.gz
tar xvfz kibana-5.6.1-linux-x86_64.tar.gz
ln -s kibana-5.6.1-linux-x86_64 kibana
cd kibana
bin/kibana
# background run
nohup bin/kibana &

Logstash 설치

cd ~/local
wget https://artifacts.elastic.co/downloads/logstash/logstash-5.6.1.tar.gz
tar xvfz logstash-5.6.1.tar.gz
ln -s logstash-5.6.1 logstash
cd logstash
mkdir logconf
vi logconf/nginx.conf

*

input {
    file {
        path => "/var/log/nginx/access.log"
        start_position => beginning
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    elasticsearch {}
}

Filebeat with logstash

cd ~/local
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.1-linux-x86_64.tar.gz
tar xvfz filebeat-5.6.1-linux-x86_64.tar.gz
ln -s filebeat-5.6.1-linux-x86_64 filebeat
cd filebeat
# elasticsearch 부분 #으로 주석 처리
  # output.elasticsearch:
    #hosts: ["localhost:9200"]
# logstash 부분 # 주석 해제
  output.logstash:
    hosts: ["localhost:5044"]

# filebeat.yml 내용 중 로그 위치 변경 `/var/log/nginx/*.log`
input {
  beats {
    port => 5044
  }
}

실행

./filebeat -e -c filebeat.yml
echo "nohup ./filebeat -e -c filebeat.yml &" > start.sh
chmod +x start.sh
./start.sh

Kibana 통계

시각화(Visualize)

대시보드 만들기

part 2

Logstash

    # params
    if [request] =~ "\?" {
        kv {
            field_split => "&"
            source => "querystring"
            include_keys => [ "query", "redirectUrl" ]
            prefix => "param_"
        }
    }

geo_point

lv,region_addr,latitude,longitude,cnt
1,강원,37.88532579,127.729829,7

Kibana

elasticsearch

Kibana 인증 with nginx

htpasswd 설치

sudo yum install httpd-tools -y
sudo htpasswd -c /etc/nginx/htpasswd.users kibanaadmin

nginx 설정 추가

sudo vi /etc/nginx/nginx.conf

Kibana with PM2

참고

What Else?