ELK

장점

사전 준비

sudo add-apt-repository ppa:openjdk-r/ppa

sudo apt-get update
sudo apt-get install openjdk-8-jdk -y

nginx 설치(샘플용)

AWS 포트 설정

설치

Elasticsearch 설치

mkdir ~/local
cd ~/local
wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.1/elasticsearch-2.4.1.tar.gz
tar xvfz elasticsearch-2.4.1.tar.gz
ln -s elasticsearch-2.4.1 elasticsearch
cd elasticsearch
vi config/elasticsearch.yml
  # `# network.host: 192.168.0.1`의 주석을 풀고 `network.host: 0.0.0.0`으로 변경
  # 모든 IP에서 접근 가능
bin/elasticsearch -d
  # 데몬(백그라운드)로 실행. 옵션 -d를 빼면 터미널 접속해 있는 동안만 실행

Kibana 설치

cd ~/local
wget https://download.elastic.co/kibana/kibana/kibana-4.6.1-linux-x86_64.tar.gz
tar xvfz kibana-4.6.1-linux-x86_64.tar.gz
ln -s kibana-4.6.1-linux-x86_64 kibana
cd kibana
bin/kibana
# background run
nohup bin/kibana &

Logstash 설치

cd ~/local
wget https://download.elastic.co/logstash/logstash/logstash-2.4.0.tar.gz
tar xvfz logstash-2.4.0.tar.gz
ln -s logstash-2.4.0 logstash
cd logstash
mkdir logconf
vi logconf/nginx.conf

logconf/nginx.conf

input {
    file {
        path => "/var/log/nginx/access.log"
        start_position => beginning
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    elasticsearch {}
    stdout {}
}

Kibana 통계

시각화(Visualize)

대시보드 만들기

part 2

Logstash

    # params
    if [request] =~ "\?" {
        kv {
            field_split => "&"
            source => "querystring"
            include_keys => [ "query", "redirectUrl" ]
            prefix => "param_"
        }
    }

Kibana

elasticsearch

Filebeat with logstash

cd ~/local
wget https://download.elastic.co/beats/filebeat/filebeat-1.3.1-x86_64.tar.gz
ln -s filebeat-1.3.1-x86_64 filebeat
cd filebeat
# elasticsearch 부분 #으로 주석 처리
  # elasticsearch:
    #hosts: ["localhost:9200"]
# logstash 부분 # 주석 해제
  logstash:
    hosts: ["localhost:5044"]

# filebeat.yml 내용 중 로그 위치 변경 `/var/log/nginx/*.log`
input {
  beats {
    port => 5044
  }
}
filter {
    grok {
        match => [
            "message", "%{COMBINEDAPACHELOG}",
            "message", "%{COMMONAPACHELOG}"
        ]
    }
    geoip {
        source => "clientip"
    }
}
output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
./filebeat -e -c filebeat.yml
echo "nohup ./filebeat -e -c filebeat.yml &" > start.sh
chmod +x start.sh
./start.sh

ELK with PM2

kibana 인증 with nginx

sudo vi /etc/nginx/nginx.conf

참고

What Else?