Elasticsearch

플러그인 설치

2.x 버전

데이터 구조 및 입출력

RDB Elasticsearch
Database Index
Table Type
Record Document
Column Field
Schema Mapping
Function Elasticsearch
HTTP method
SQL
Create POST INSERT
Read GET SELECT
Update PUT UPDATE
Delete DELETE DELETE

데이터 입력

curl -XPOST http://localhost:9200/books/book/1 -d '
{
  "title" : "elasticsearch guide",
  "author" : "Kim",
  "date" : "2016-05-22",
  "pages" : 250
}
'

데이터 조회

curl -XGET http://localhost:9200/books/book/1
# 또는
curl -XGET http://localhost:9200/books/book/1\?pretty

데이터 갱신

curl -XPOST http://localhost:9200/books/book/1 -d '
{
  "title" : "elasticsearch guide",
  "author" : ["Kim", "Heo"],
  "date" : "2016-05-22",
  "pages" : 300
}
'

데이터 삭제

curl -XDELETE http://localhost:9200/books/book/1

데이터 배치 입력(_bulk API)

{ "delete" : { "_index" : "books", "_type" : "book", "_id" : "1" } }
{ "update" : { "_index" : "books", "_type" : "book", "_id" : "2" } }
{ "doc" : { "date" : "2014-05-01" } }
{ "create" : { "_index" : "books", "_type" : "book", "_id" : "3" } }
{ "title" : "Elasticsearch Guide II", "author" : "Park", "pages" : 400 }
curl -XPOST http://localhost:9200/_bulk?pretty --data-binary @data.txt

index list

curl localhost:9200/_cat/indices?v

검색

검색 API

어그리게이션(aggregation)

글로벌 어그리게이션

필터 어그리게이션

누락(missing) 어그리게이션

텀 어그리게이션

curl 'localhost:9200/hotels/_search?pretty' -d '
{
  "aggs" : {
    "term_stars" : {
      "terms" : {
        "field" : "stars",
        "order" : { "_term" : "desc" }
      },
      "aggs" : {
        "avg_price" : {
          "avg" : { "field" : "price" }
        }
      }
    }
  }
}'
curl 'localhost:9200/hotels/_search?pretty' -d '
{
  "aggs" : {
    "term_stars" : {
      "terms" : {
        "field" : "stars",
        "order" : { "avg_price" : "asc" }
      },
      "aggs" : {
        "avg_price" : {
          "avg" : { "field" : "price" }
        }
      }
    }
  }
}'

범위, 날짜 범위 어그리게이션

curl 'localhost:9200/hotels/_search?pretty' -d '
{
  "aggs" : {
    "range_room" : {
      "range" : {
        "field" : "rooms",
        "ranges" : [{"to":500}, {"from":500, "to":1000}, {"from":1000}]
      },
      "aggs" : {
        "avg_price" : {
          "avg" : { "field" : "price" }
        }
      }
    }
  }
}'
curl 'localhost:9200/hotels/_search?pretty' -d '
{
  "aggs" : {
    "date_r_checkin" : {
      "date_range" : {
        "field" : "checkin",
        "format" : "yyyy-MM-dd hh:mm:ss",
        "ranges" : [{"to": "2014-03-05 12:30:45"}, {"from": "2014-03-05 12:30:45"}]
      }
    }
  }
}'

히스토그램

curl 'localhost:9200/hotels/_search?pretty' -d '
{
  "aggs" : {
    "histo_rooms" : {
      "histogram" : {
        "field" : "rooms",
        "interval" : 500
      }
    }
  }
}'
curl 'localhost:9200/hotels/_search?pretty' -d '
{
  "aggs" : {
    "histo_rooms" : {
      "histogram" : {
        "field" : "rooms",
        "interval" : 300,
        "min_doc_count" : 0
      }
    }
  }
}'

위치, 거리

curl 'localhost:9200/hotels/_search?pretty' -d '
{
  "aggs" : {
    "geo_location" : {
      "geo_distance" : {
        "field" : "location",
        "origin" : "37.52, 126.98",
        "distance_type" : "plane",
        "unit" : "km",
        "ranges" : [{"to":3},{"from":3, "to":6},{"from":6, "to":9},{"from":9}]
      }
    }
  }
}'

질의(QueryDSL)

curl 'localhost:9200/books/_search?pretty' -d '
{
  "query" : {
    "term" : {
      "title" : "prince"
    }
  }
}
'
curl 'localhost:9200/books/_search?pretty' -d '
{
  "query" : {
    "bool" : {
      "filter" : {
        "term" : {
          "title" : "prince"
        }
      }
    }
  }
}
'

Query

curl 'localhost:9200/books/_search?pretty' -d '
{
  "query" : {
    "query_string" : {
      "query" : "prince king",
      "default_field" : "plot",
      "default_operator" : "and"
    }
  }
}'
curl 'localhost:9200/books/_search?pretty' -d '
{
  "query" : {
    "range" : {
      "written" : {
        "gte" : "1600-01-01",
        "lt" : "1699-12-31"
      }
    }
  }
}'

매핑

curl 'localhost:9200/books/_mapping?pretty'

내장필드

curl -XDELETE 'http://localhost:9200/books'

curl -XPUT 'http://localhost:9200/books' -d '
{
  "mappings" : {
    "book" : {
      "_source" : { "enabled"  : false }
    }
  }
}'

curl -XPOST localhost:9200/_bulk --data-binary @5_1_books.json

curl 'http://localhost:9200/books/_search?q=prince&pretty'
curl -XPUT 'http://localhost:9200/books' -d '
{
  "mappings" : {
    "book" : {
      "_source" : {
        "includes" : ["title", "author", "category"]
      }
    }
  }
}'
curl -XPUT 'http://localhost:9200/books' -d '
{
  "mappings" : {
    "book" : {
      "_source" : {
        "excludes" : ["p*"]
      }
    }
  }
}'

데이터 타입

이름 설명 기본값
store 필드값 저장 여부 false
index 분석기 적용 여부 analyzed, not_analyzed, no .
boost 필드 가중치 1.0
null_value 필드 없는 경우 기본값 지정 .
analyzer 분석기 지정 .
index_analyzer 데이터 색인에 사용될 분석기 지정 .
search_analyzer 문자열 검색에 사용될 분석기 지정 .
include_in_all _all 매핑 필드 적용된 경우 색인 여부 지정 .
ignore_above 지정값보다 큰 크기의 문자열 색인 제외 .
curl -XPUT 'localhost:9200/test_nums' -d '
{
  "mappings" : {
    "test_num" : {
      "properties" : {
        "num_val" : { "type" : "integer", "ignore_malformed" : true }
      }
    }
  }
}'
curl -XPUT 'localhost:9200/test_nums/test_num/1' -d '
{
  "num_val": "hello"
}'
curl 'localhost:9200/test_nums/test_num/1'
curl 'localhost:9200/test_nums/_search?pretty' -d '
{
  "aggs" : {
    "num_stat" : {
      "stats" : { "field" : "num_val" }
    }
  }
}'
curl -XPUT 'localhost:9200/test_geos/test_geo/1' -d '
{
  "name" : "Conrad Seoul",
  "location" : "37.525308, 126.926644"
}'
curl 'http://localhost:9200/test_geos/_search?pretty' -d '
{
  "query" : {
    "geo_bounding_box" : {
      "location" : {
        "top_left": { "lat" : 37.53, "lon" : 126.92 },
        "bottom_right" : {"lat" : 37.52, "lon" : 126.93 }
      }
    }
  }
}'

다중필드

curl 'localhost:9200/books/_search?pretty' -d '
{
  "query" : {
    "term" : { "title.raw" : "The Prince and the Pauper" }
  }
}'
curl 'localhost:9200/books/_search?pretty' -d '
{
  "fields" : [ "title", "title.tokens" ],
  "query" : {
    "term" : { "title" : "the" }
  }
}'

필드 복사

curl -XPUT 'localhost:9200/books' -d '
{
  "mappings" : {
    "book" : {
      "properties" : {
        "title" : { "type" : "string", "copy_to" : ["title_1", "title_2"] },
        "title_1" : { "type" : "string", "store" : true },
        "title_2" : { "type" : "string", "store" : true }
      }
    }
  }
}'

분석

curl -XPOST 'localhost:9200/_analyze?tokenizer=whitespace&pretty' -d 'Around the World in Eighty Days'

curl -XPOST 'localhost:9200/_analyze?tokenizer=whitespace&filters=lowercase&pretty' -d 'Around the World in Eighty Days'

curl -XPOST 'localhost:9200/_analyze?tokenizer=whitespace&filters=lowercase,stop&pretty' -d 'Around the World in Eighty Days'
curl -XPOST 'localhost:9200/books/_analyze?pretty' -d 'Around the World in Eighty Days'

분석기

curl -XPOST 'localhost:9200/books/_analyze?analyzer=standard&pretty' -d 'Around the World in Eighty Days'
echo 'in
the
world' > config/stopword_list.txt
curl -XPUT 'localhost:9200/books' -d '
{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "stop" : {
          "type" : "stop",
          "stopwords_path" : "stopword_list.txt"
        }
      }
    }
  }
}'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=stop&pretty' -d 'Around the World in Eighty Days'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=pattern&pretty' -d 'Around the World in Eighty Days'
curl -XPUT 'localhost:9200/books' -d '
{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "pattern" : {
          "type" : "pattern",
          "pattern" : "\\d"
        }
      }
    }
  }
}'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=pattern&pretty' -d 'Around the World in 80 Days'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=language&pretty' -d '삼국지(三國志)'
curl -XPUT 'localhost:9200/books' -d '
{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "language" : {
          "type" : "cjk"
        }
      }
    }
  }
}'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=language&pretty' -d '삼국지(三國志)'
curl -XPUT 'localhost:9200/books' -d '
{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "snowball" : {
          "type" : "snowball",
          "pattern" : "english"
        }
      }
    }
  }
}'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=snowball&pretty' -d 'Around the World in Eighty Days'

토크나이저

curl -XPUT 'localhost:9200/books' -d '
{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "my_analyzer" : {
          "tokenizer" : "standard"
        }
      }
    }
  }
}'

# 또는

curl -XPUT 'localhost:9200/books' -d '
{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "my_analyzer" : {
          "tokenizer" : "my_tokenizer"
        }
      },
      "tokenizer" : {
        "my_tokenizer" : {
          "type" : "standard"
        }
      }
    }
  }
}'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'Around the World in 80 Days'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'Around the World in 80 Days'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'Program/Search/Elasticsearch'
curl -XPUT 'localhost:9200/books' -d '
{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "my_analyzer" : {
          "tokenizer" : "my_hierarchy"
        }
      },
      "tokenizer" : {
        "my_hierarchy" : {
          "type" : "path_hierarchy",
          "replacement" : "$",
          "skip" : 1
        }
      }
    }
  }
}'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'Program/Search/Elasticsearch'

토큰필터

curl -XPUT 'localhost:9200/books' -d '
{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "my_analyzer" : {
          "tokenizer" : "standard",
          "filter" : ["standard"]
        }
      }
    }
  }
}'

# 또는

curl -XPUT 'localhost:9200/books' -d '
{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "my_analyzer" : {
          "tokenizer" : "standard",
          "filter" : ["my_tokenfilter"]
        }
      },
      "filter" : {
        "my_tokenfilter" : {
          "type" : "standard"
        }
      }
    }
  }
}'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'Around the World in Eighty Days'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'Around the World in Eighty Days'
curl -XPUT 'localhost:9200/books' -d '
{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "my_analyzer" : {
          "tokenizer" : "standard",
          "filter" : ["stop", "my_tokenfilter"]
        }
      },
      "filter" : {
        "my_tokenfilter" : {
          "type" : "shingle",
          "max_shingle_size" : 3,
          "min_shingle_size" : 3,
          "output_unigrams" : false,
          "token_separator" : "-",
          "filler_token" : "*"
        }
      }
    }
  }
}'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'Around the World in Eighty Days'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d "Father's Wi-Fi SmartPhone, SD3000-12-Delux"
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'Birds are flying, fishes are swimming, children are playing'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'Birds are flying, fishes are swimming, children are playing'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'Birds are flying, fishes are swimming, children are playing'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'The Quick Rabbit Jumped'
curl -XPUT 'localhost:9200/books' -d '
{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "my_analyzer" : {
          "tokenizer" : "standard",
          "filter" : ["snowball", "lowercase", "my_filter"]
        }
      },
      "filter" : {
        "my_filter" : {
          "type" : "synonym",
          "synonyms_path" : "synonym.txt"
        }
      }
    }
  }
}'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'The Quick Rabbit Jumped'
curl -XPUT 'localhost:9200/books' -d '
{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "my_analyzer" : {
          "tokenizer" : "standard",
          "filter" : ["my_filter"]
        }
      },
      "filter" : {
        "my_filter" : {
          "type" : "dictionary_decompounder",
          "word_list" : ["base", "ball"]
        }
      }
    }
  }
}'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'I play baseball and basketball'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'Around the World in Eighty Days'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'Around the World in Eighty Days'
cd $ELASTICSEARCH_HOME
mkdir -p ../config/hunspell/en_US
cp en_US.* ../config/hunspell/en_US

mkdir -p ../config/hunspell/ko_KR
cp ko_KR.* ../config/hunspell/ko_KR

curl -XPUT 'localhost:9200/books' -d '
{
  "settings" : {
    "analysis" : {
      "analyzer" : {
        "my_analyzer" : {
          "tokenizer" : "standard",
          "filter" : ["my_filter"]
        }
      },
      "filter" : {
        "my_filter" : {
          "type" : "hunspell",
          "locale" : "ko_KR"
        }
      }
    }
  }
}'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d '동해물과 백두산이 마르고 닳도록'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d '삼국지(三國志)'
curl -XPOST 'localhost:9200/books/_analyze?analyzer=my_analyzer&pretty' -d 'Around the World in Eighty Days'

한글 형태소 분석기

#!/usr/bin/env bash

ES='http://localhost:9200'
ESIDX='seunjeon-idx'

curl -XDELETE "${ES}/${ESIDX}?pretty"
sleep 1
curl -XPUT "${ES}/${ESIDX}/?pretty" -d '{
  "settings" : {
    "index":{
      "analysis":{
        "analyzer":{
          "korean":{
            "type":"custom",
            "tokenizer":"seunjeon_default_tokenizer"
          }
        },
        "tokenizer": {
          "seunjeon_default_tokenizer": {
            "type": "seunjeon_tokenizer",
            "index_eojeol": false,
            "user_words": ["낄끼+빠빠,-100", "c\\+\\+", "어그로", "버카충", "abc마트"]
          }
        }
      }
    }
  }
}'

sleep 1

echo "# 삼성/N 전자/N"
curl -XGET "${ES}/${ESIDX}/_analyze?analyzer=korean&pretty" -d '삼성전자'

echo "# 빠르/V 지/V"
curl -XGET "${ES}/${ESIDX}/_analyze?analyzer=korean&pretty" -d '빨라짐'

echo "# 슬프/V"
curl -XGET "${ES}/${ESIDX}/_analyze?analyzer=korean&pretty" -d '슬픈'

echo "# 새롭/V 사전/N 생성/N"
curl -XGET "${ES}/${ESIDX}/_analyze?analyzer=korean&pretty" -d '새로운사전생성'

echo "# 낄끼/N 빠빠/N c++/N"
curl -XGET "${ES}/${ESIDX}/_analyze?analyzer=korean&pretty" -d '낄끼빠빠 c++'
curl -XPOST 'localhost:9200/${ES}/_analyze?analyzer=korean&pretty' -d '동해물과 백두산이 마르고 닳도록'

부분삭제

curl -XDELETE 'http://localhost:9200/twitter/tweet/_delete_by_query?q=user:kimchy'

#or
curl -XPOST 'http://localhost:9200/twitter/tweet/_delete_by__query' -d '
{
  "query" : {
    "term" : {
      "user" : "kimchy"
    }
  }
}'

cluster

ubuntu plugin

참고

What Else?