본문 바로가기
Elastic Stack

Elastiscearch, Kibana, Metricbeat 구축 Using Docker Compose (Single Node&Multi Node)

by study4me 2025. 6. 24.
반응형

1. 참고 페이지

2. 환경 정보

구분 서비스 구성 비고
Single Node EC2 c5.xlarge 1대 하나의 EC2에 Elasticsearch, Kibana, Metricbeat 컨테이너 배포
Multi Node EC2 c5.large 4대 3대의 EC2에 Elasticsearch, Metircbeat 컨테이너 배포, 1대의 EC2에 Kibana 배포
공통 OS Amazon linux 2023 -
공통 EBS EC2 당 3개 os, data, log 용도
공통 IAM Role S3 권한, EC2에 Attach Elasticsearch Snapshot API Backup을 위함
공통 S3 Bucket 1개 Elasticsearch Snapshot API Backup을 위함

* CPU 대비 Memory 사용률이 높으므로 c5보다 m5 시리즈 추천

3. 아키텍처

본 게시글은 EC2 내부 구성을 어떻게 했는지에 대한 기록으로 일부 아키텍처는 생략했다.

 

위 아키텍처에서 EC2에 해당하는 부분을 상세하게 보면 아래와 같다.

Single Node 구성(개발), Multi Node 구성(운영)

 

4. 공통 환경 구축

각 EC2에서 다음 명령어를 실행한다. hostname, hosts, 디렉토리 구성은 각 EC2마다 적절하게 실행한다.

1) Date 변경

ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime
date

2) Swap Off

swapoff -a
sed -i '/ swap / s/^/#/' /etc/fstab

3) 가상 메모리 설정 변경

sysctl vm.max_map_count

cat <<EOF | sudo tee /etc/sysctl.d/elasticsearch.conf
vm.max_map_count=262144 
EOF

sysctl --system
sysctl vm.max_map_count

4) EBS 마운트

# 디스크 조회
lsblk
## NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
## (생략)
## nvme1n1       259:1    0   50G  0 disk
## nvme2n1       259:2    0   10G  0 disk

# 파일 시스템 유무 확인(반환값: data)
file -s /dev/nvme1n1
file -s /dev/nvme2n1


# 파일 시스템 생성
mkfs -t xfs /dev/nvme1n1
mkfs -t xfs /dev/nvme2n1


# 마운트 포인트 생성
mkdir /data001
mkdir /logs001

# 마운트
mount /dev/nvme1n1 /data001
mount /dev/nvme2n1 /logs001

# 영구 마운트
## fstab 백업
cp /etc/fstab /etc/fstab.orig
## UUID 추출
UUID1=$(blkid -s UUID -o value /dev/nvme1n1)
UUID2=$(blkid -s UUID -o value /dev/nvme2n1)
echo $UUID1
echo $UUID2
echo "UUID=$UUID1     /data001  xfs  defaults,nofail 0 2" >> /etc/fstab
echo "UUID=$UUID2     /logs001  xfs  defaults,nofail 0 2" >> /etc/fstab
cat /etc/fstab

# 재부팅 후 확인
reboot
lsblk

5) hostname 설정

# 개발
hostnamectl set-hostname EC2-DEV-ELASTIC-STACK-01

# 운영
hostnamectl set-hostname EC2-PROD-ELASTICSEARCH-01
hostnamectl set-hostname EC2-PROD-ELASTICSEARCH-02
hostnamectl set-hostname EC2-PROD-ELASTICSEARCH-03
hostnamectl set-hostname EC2-PROD-KIBANA-01

6) hosts 수정

cat /etc/hosts

# 개발
cat <<EOF >> /etc/hosts
10.1.x.x EC2-DEV-ELASTIC-STACK-01
EOF

# 운영
cat <<EOF >> /etc/hosts
10.1.x.x EC2-PROD-ELASTICSEARCH-01
10.1.x.x EC2-PROD-ELASTICSEARCH-02
10.1.x.x EC2-PROD-ELASTICSEARCH-03
10.1.x.x EC2-PROD-KIBANA-01
EOF

7) Docker 설치 (Amazon Linux 2023)

# 기존 환경 제거
dnf remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine \
                  docker-selinux \
                  docker-engine-selinux \
                  podman \
                  runc
# 설치
dnf -y install dnf-plugins-core
# amazon linux 2023은 공식 docker에서 지원하지 않아 centos의 repo를 빌려 쓴다.
dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sed -i 's/\$releasever/8/g' /etc/yum.repos.d/docker-ce.repo
dnf -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
systemctl start docker
systemctl enable docker
systemctl status docker --no-pager 
docker version

# Docker Log 설정
cat <<EOF >> /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "20m",
    "max-file": "5"
  }
}
EOF

8) User 생성 및 권한 부여

adduser svc
usermod -aG docker svc
usermod -aG ec2-user svc

visudo (저장&나가기 ctrl + s, ctrl + x)
svc  ALL=(ALL)  NOPASSWD:ALL

9) Docker에서 영구 보존할 디렉토리 생성

# Elasticsearch 컨테이너가 배포되는 EC2에서 실행
mkdir -p /data001/elasticsearch/config/certs
mkdir -p /data001/elasticsearch/data
chown -R 1000:1000 /data001/elasticsearch
chown root:root /data001/elasticsearch/config/certs
chmod -R 774 /data001/elasticsearch
chmod 750 /data001/elasticsearch/config/certs
mkdir -p /logs001/elasticsearch/logs
chown -R 1000:1000 /logs001/elasticsearch
chmod -R 774 /logs001/elasticsearch

# Kibana 컨테이너가 배포되는 EC2에서 실행
mkdir -p /data001/kibana/config/certs
mkdir -p /data001/kibana/data
chown -R 1000:1000 /data001/kibana
chown root:root /data001/kibana/config/certs
chmod -R 774 /data001/kibana
chmod 750 /data001/kibana/config/certs
mkdir -p /logs001/kibana/logs
chown -R 1000:1000 /logs001/kibana
chmod -R 774 /logs001/kibana

# MetricBeat 컨테이너가 배포되는 EC2에서 실행
mkdir -p /data001/metricbeat/data
mkdir -p /data001/metricbeat/config/certs

5. Docker Compose로 배포

1) 작업 디렉토리 생성

sudo su - svc
mkdir ~/docker_compose

2) docker compose yml 파일

/home/svc/docker_compose 하위에 아래와 같이 파일을 생성한다.

환경 EC2 Name 필요한 파일
개발 EC2-DEV-ELASTIC-STACK-01 .env, createcerts.yml, setup.yml, elasticsearch01.yml, kibana.yml, metricbeat01.yml
운영 EC2-PROD-ELASTICSEARCH-01 . env, createcerts.yml, setup.yml, elasticsearch01.yml, metricbeat01.yml
운영 EC2-PROD-ELASTICSEARCH-02 .env, elasticsearch02.yml, metricbeat02.yml
운영 EC2-PROD-ELASTICSEARCH-03 . env, elasticsearch03.yml, metricbeat03.yml
운영 EC2-PROD-KIBANA-01 .env, kibana.yml

 

개발

.env 
# Password for the 'elastic' user (at least 6 characters)
ELASTIC_PASSWORD=E1@sticPW

# Password for the 'kibana_system' user (at least 6 characters)
KIBANA_PASSWORD=E1@sticPW

# Version of Elastic products
# STACK_VERSION=8.17.5

# Set the cluster name
CLUSTER_NAME=dev-cluster

# Set to 'basic' or 'trial' to automatically start the 30-day trial
LICENSE=basic
#LICENSE=trial

# Port to expose Elasticsearch HTTP API to the host
ES_PORT=9200
#ES_PORT=127.0.0.1:9200

# Port to expose Kibana to the host
KIBANA_PORT=5601
#KIBANA_PORT=80

# Increase or decrease based on the available host memory (in bytes)
# 2.5GiB=2684354560
#   2GiB=2147483648
# 2.5iGB일 때 힙메모리 자동으로 1.25GiB(1342177280) 할당됨
#   2GiB일 때 힙메모리 자동으로    1GiB(1073741824) 할당된 것으로 기억
ELASTIC_MEM_LIMIT=2684354560
KIBANA_MEM_LIMIT=2147483648
METRICBEAT_MEM_LIMIT=536870912

# Project namespace (defaults to the current folder name if not set)
COMPOSE_PROJECT_NAME=dev


ELASTIC_IMAGE=docker.elastic.co/elasticsearch/elasticsearch:8.17.5
KIBANA_IMAGE=docker.elastic.co/kibana/kibana:8.17.5
METRICBEAT_IMAGE=docker.elastic.co/beats/metricbeat:8.17.5
ES01_IP=10.1.x.x
KIBANA_IP=10.1.x.x
ES01_HOSTNAME=EC2-DEV-ELASTIC-STACK-01
KIBANA_HOSTNAME=EC2-DEV-ELASTIC-STACK-01
DOMAIN=kibana-dev.ohdi.co.kr
createcerts.yml 
services:
  createcerts:
    image: ${ELASTIC_IMAGE}
    volumes:
      - /data001/elasticsearch/config/certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - ${ES01_HOSTNAME}\n"\
          "    ip:\n"\
          "      - ${ES01_IP}\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Certificate creation completed!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120
setup.yml 
services:
  setup:
    image: ${ELASTIC_IMAGE}
    volumes:
      - /data001/elasticsearch/config/certs:/usr/share/elasticsearch/config/certs
    extra_hosts:
      - "${ES01_HOSTNAME}=${ES01_IP}"
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://${ES01_HOSTNAME}:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://${ES01_HOSTNAME}:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
elasticsearch01.yml 
services:
  es01:
    image: ${ELASTIC_IMAGE}
    restart: unless-stopped
    volumes:
      - /data001/elasticsearch/config/certs:/usr/share/elasticsearch/config/certs
      - /data001/elasticsearch/data:/usr/share/elasticsearch/data
      - /logs001/elasticsearch/logs:/usr/share/elasticsearch/logs
    ports:
      - ${ES_PORT}:9200
      - 9300:9300
    environment:
      - network.host=0.0.0.0
      - network.publish_host=${ES01_HOSTNAME}
      - node.roles=master,data_content,data_hot,ingest,transform,remote_cluster_client
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - discovery.type=single-node
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - xpack.ml.use_auto_machine_memory_percent=true
    mem_limit: ${ELASTIC_MEM_LIMIT}
    extra_hosts:
      - "${ES01_HOSTNAME}=${ES01_IP}"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nproc:
        soft: 4096
        hard: 4096
      nofile:
        soft: 65535
        hard: 65535
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://${ES01_HOSTNAME}:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
kibana.yml 
services:
  kibana:
    image: ${KIBANA_IMAGE}
    restart: unless-stopped
    volumes:
      - /data001/kibana/config/certs:/usr/share/kibana/config/certs
      - /data001/kibana/data:/usr/share/kibana/data
      - /logs001/kibana/logs:/usr/share/kibana/logs
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVER_PUBLICBASEURL=https://${DOMAIN}:5601
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=["https://${ES01_HOSTNAME}:9200"]
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - TELEMETRY_ENABLED=false
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=xxxxxxxxxxxxxxxxxxxxxxx
      - XPACK_REPORTING_ENCRYPTIONKEY=xxxxxxxxxxxxxxxxxxxxxxx
      - XPACK_SECURITY_ENCRYPTIONKEY=xxxxxxxxxxxxxxxxxxxxxxx
    mem_limit: ${KIBANA_MEM_LIMIT}
    extra_hosts:
      - "${ES01_HOSTNAME}=${ES01_IP}"
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
metircbeat01.yml 
services:
  metricbeat01:
    image: ${METRICBEAT_IMAGE}
    restart: unless-stopped
    user: root
    volumes:
      - /data001/metricbeat/config/certs:/usr/share/metricbeat/certs
      - /data001/metricbeat/data:/usr/share/metricbeat/data
      - "/data001/metricbeat/config/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
      - "/proc:/hostfs/proc:ro"
      - "/:/hostfs:ro"
    environment:
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://${ES01_HOSTNAME}:9200
      - KIBANA_HOSTS=http://${KIBANA_HOSTNAME}:5601
    mem_limit: ${METRICBEAT_MEM_LIMIT}
    extra_hosts:
      - "${ES01_HOSTNAME}=${ES01_IP}"
      - "${KIBANA_HOSTNAME}=${KIBANA_IP}"

 

운영

.env 
# Password for the 'elastic' user (at least 6 characters)
ELASTIC_PASSWORD=E1@sticPW

# Password for the 'kibana_system' user (at least 6 characters)
KIBANA_PASSWORD=E1@sticPW

# Version of Elastic products
# STACK_VERSION=8.17.5

# Set the cluster name
CLUSTER_NAME=prod-cluster

# Set to 'basic' or 'trial' to automatically start the 30-day trial
LICENSE=basic
#LICENSE=trial

# Port to expose Elasticsearch HTTP API to the host
ES_PORT=9200
#ES_PORT=127.0.0.1:9200

# Port to expose Kibana to the host
KIBANA_PORT=5601
#KIBANA_PORT=80

# Increase or decrease based on the available host memory (in bytes)
# 2.5GiB=2684354560
#   2GiB=2147483648
# 2.5iGB일 때 힙메모리 자동으로 1.25GiB(1342177280) 할당됨
#   2GiB일 때 힙메모리 자동으로    1GiB(1073741824) 할당된 것으로 기억
ELASTIC_MEM_LIMIT=2684354560
KIBANA_MEM_LIMIT=2147483648
METRICBEAT_MEM_LIMIT=536870912

# Project namespace (defaults to the current folder name if not set)
COMPOSE_PROJECT_NAME=prod


ELASTIC_IMAGE=docker.elastic.co/elasticsearch/elasticsearch:8.17.5
KIBANA_IMAGE=docker.elastic.co/kibana/kibana:8.17.5
METRICBEAT_IMAGE=docker.elastic.co/beats/metricbeat:8.17.5
ES01_IP=10.1.x.x
ES02_IP=10.1.x.x
ES03_IP=10.1.x.x
KIBANA_IP=10.1.x.x
ES01_HOSTNAME=EC2-PROD-ELASTICSEARCH-01
ES02_HOSTNAME=EC2-PROD-ELASTICSEARCH-02
ES03_HOSTNAME=EC2-PROD-ELASTICSEARCH-03
KIBANA_HOSTNAME=EC2-PROD-KIBANA-01
DOMAIN=kibana-prod.ohdi.co.kr
createcerts.yml 
services:
  createcerts:
    image: ${ELASTIC_IMAGE}
    volumes:
      - /data001/elasticsearch/config/certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - ${ES01_HOSTNAME}\n"\
          "    ip:\n"\
          "      - ${ES01_IP}\n"\
          "  - name: es02\n"\
          "    dns:\n"\
          "      - ${ES02_HOSTNAME}\n"\
          "    ip:\n"\
          "      - ${ES02_IP}\n"\
          "  - name: es03\n"\
          "    dns:\n"\
          "      - ${ES03_HOSTNAME}\n"\
          "    ip:\n"\
          "      - ${ES03_IP}\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Certificate creation completed!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120
setup.yml 
services:
  setup:
    image: ${ELASTIC_IMAGE}
    volumes:
      - /data001/elasticsearch/config/certs:/usr/share/elasticsearch/config/certs
    extra_hosts:
      - "${ES01_HOSTNAME}=${ES01_IP}"
      - "${ES02_HOSTNAME}=${ES02_IP}"
      - "${ES03_HOSTNAME}=${ES03_IP}"
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://${ES01_HOSTNAME}:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://${ES01_HOSTNAME}:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
elasticsearch01.yml 
services:
  es01:
    image: ${ELASTIC_IMAGE}
    restart: unless-stopped
    volumes:
      - /data001/elasticsearch/config/certs:/usr/share/elasticsearch/config/certs
      - /data001/elasticsearch/data:/usr/share/elasticsearch/data
      - /logs001/elasticsearch/logs:/usr/share/elasticsearch/logs
    ports:
      - ${ES_PORT}:9200
      - 9300:9300
    environment:
      - network.host=0.0.0.0
      - network.publish_host=${ES01_HOSTNAME}
      - node.roles=master,data_content,data_hot,ingest,transform,remote_cluster_client
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=${ES02_HOSTNAME},${ES03_HOSTNAME}
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - xpack.ml.use_auto_machine_memory_percent=true
    mem_limit: ${ELASTIC_MEM_LIMIT}
    extra_hosts:
      - "${ES01_HOSTNAME}=${ES01_IP}"
      - "${ES02_HOSTNAME}=${ES02_IP}"
      - "${ES03_HOSTNAME}=${ES03_IP}"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nproc:
        soft: 4096
        hard: 4096
      nofile:
        soft: 65535
        hard: 65535
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://${ES01_HOSTNAME}:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

 

elasticsearch02.yml 
services:
  es02:
    image: ${ELASTIC_IMAGE}
    restart: unless-stopped
    volumes:
      - /data001/elasticsearch/config/certs:/usr/share/elasticsearch/config/certs
      - /data001/elasticsearch/data:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
      - 9300:9300
    environment:
      - network.host=0.0.0.0
      - network.publish_host=${ES02_HOSTNAME}
      - node.roles=master,data_content,data_hot,ingest,transform,remote_cluster_client
      - node.name=es02
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=${ES01_HOSTNAME},${ES03_HOSTNAME}
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es02/es02.key
      - xpack.security.http.ssl.certificate=certs/es02/es02.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es02/es02.key
      - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - xpack.ml.use_auto_machine_memory_percent=true
    mem_limit: ${ELASTIC_MEM_LIMIT}
    extra_hosts:
      - "${ES01_HOSTNAME}=${ES01_IP}"
      - "${ES02_HOSTNAME}=${ES02_IP}"
      - "${ES03_HOSTNAME}=${ES03_IP}"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nproc:
        soft: 4096
        hard: 4096
      nofile:
        soft: 65535
        hard: 65535
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://${ES02_HOSTNAME}:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
elasticsearch03.yml 
services:
  es03:
    image: ${ELASTIC_IMAGE}
    restart: unless-stopped
    volumes:
      - /data001/elasticsearch/config/certs:/usr/share/elasticsearch/config/certs
      - /data001/elasticsearch/data:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
      - 9300:9300
    environment:
      - network.host=0.0.0.0
      - network.publish_host=${ES03_HOSTNAME}
      - node.roles=master,data_content,data_hot,ingest,transform,remote_cluster_client
      - node.name=es03
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=${ES01_HOSTNAME},${ES02_HOSTNAME}
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es03/es03.key
      - xpack.security.http.ssl.certificate=certs/es03/es03.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es03/es03.key
      - xpack.security.transport.ssl.certificate=certs/es03/es03.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - xpack.ml.use_auto_machine_memory_percent=true
    mem_limit: ${ELASTIC_MEM_LIMIT}
    extra_hosts:
      - "${ES01_HOSTNAME}=${ES01_IP}"
      - "${ES02_HOSTNAME}=${ES02_IP}"
      - "${ES03_HOSTNAME}=${ES03_IP}"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nproc:
        soft: 4096
        hard: 4096
      nofile:
        soft: 65535
        hard: 65535
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://${ES03_HOSTNAME}:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
kibana.yml 
services:
  kibana:
    image: ${KIBANA_IMAGE}
    restart: unless-stopped
    volumes:
      - /data001/kibana/config/certs:/usr/share/kibana/config/certs
      - /data001/kibana/data:/usr/share/kibana/data
      - /logs001/kibana/logs:/usr/share/kibana/logs
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVER_PUBLICBASEURL=https://${DOMAIN}:5601
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=["https://${ES01_HOSTNAME}:9200","https://${ES02_HOSTNAME}:9200","https://${ES03_HOSTNAME}:9200"]
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - TELEMETRY_ENABLED=false
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=bd8627f802a2377a6a3414f4b582ea5a
      - XPACK_REPORTING_ENCRYPTIONKEY=ab16ff0f83020f6db701572912eb6790
      - XPACK_SECURITY_ENCRYPTIONKEY=76d039217db056018e924cce7d105f5c
    mem_limit: ${KIBANA_MEM_LIMIT}
    extra_hosts:
      - "${ES01_HOSTNAME}=${ES01_IP}"
      - "${ES02_HOSTNAME}=${ES02_IP}"
      - "${ES03_HOSTNAME}=${ES03_IP}"
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
metircbeat01.yml 
services:
  metricbeat01:
    image: ${METRICBEAT_IMAGE}
    restart: unless-stopped
    user: root
    volumes:
      - /data001/metricbeat/config/certs:/usr/share/metricbeat/certs
      - /data001/metricbeat/data:/usr/share/metricbeat/data
      - "/data001/metricbeat/config/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
      - "/proc:/hostfs/proc:ro"
      - "/:/hostfs:ro"
    environment:
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://${ES01_HOSTNAME}:9200
      - KIBANA_HOSTS=http://${KIBANA_HOSTNAME}:5601
    mem_limit: ${METRICBEAT_MEM_LIMIT}
    extra_hosts:
      - "${ES01_HOSTNAME}=${ES01_IP}"      
      - "${KIBANA_HOSTNAME}=${KIBANA_IP}"
metircbeat02.yml 
services:
  metricbeat02:
    image: ${METRICBEAT_IMAGE}
    restart: unless-stopped
    user: root
    volumes:
      - /data001/metricbeat/config/certs:/usr/share/metricbeat/certs
      - /data001/metricbeat/data:/usr/share/metricbeat/data
      - "/data001/metricbeat/config/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
      - "/proc:/hostfs/proc:ro"
      - "/:/hostfs:ro"
    environment:
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://${ES02_HOSTNAME}:9200
      - KIBANA_HOSTS=http://${KIBANA_HOSTNAME}:5601
    mem_limit: ${METRICBEAT_MEM_LIMIT}
    extra_hosts:
      - "${ES02_HOSTNAME}=${ES02_IP}"      
      - "${KIBANA_HOSTNAME}=${KIBANA_IP}"
metircbeat03.yml 
services:
  metricbeat03:
    image: ${METRICBEAT_IMAGE}
    restart: unless-stopped
    user: root
    volumes:
      - /data001/metricbeat/config/certs:/usr/share/metricbeat/certs
      - /data001/metricbeat/data:/usr/share/metricbeat/data
      - "/data001/metricbeat/config/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
      - "/proc:/hostfs/proc:ro"
      - "/:/hostfs:ro"
    environment:
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://${ES03_HOSTNAME}:9200
      - KIBANA_HOSTS=http://${KIBANA_HOSTNAME}:5601
    mem_limit: ${METRICBEAT_MEM_LIMIT}
    extra_hosts:
      - "${ES03_HOSTNAME}=${ES03_IP}"
      - "${KIBANA_HOSTNAME}=${KIBANA_IP}"

3) Metricbeat Config 파일 생성

sudo su - root
vi /data001/metricbeat/config/metricbeat.yml

아래 metricbeat.yml에서 es01가 작성된 부분을 각 서버에 맞게 수정이 필요하다.
예를 들어 elasticsearch02.yml을 배포한 서버에서는 es02로 수정해 준다.
(metricbeat.modules와 output.elasticsearch 하위 부분에 해당)

metricbeat.yml
 
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false


metricbeat.modules:
- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ${ELASTIC_HOSTS}
  ssl.certificate_authorities: "certs/ca/ca.crt"
  ssl.certificate: "certs/es01/es01.crt"
  ssl.key: "certs/es01/es01.key"
  username: ${ELASTIC_USER}
  password: ${ELASTIC_PASSWORD}
  ssl.enabled: true

- module: kibana
  metricsets:
    - stats
  period: 10s
  hosts: ${KIBANA_HOSTS}
  username: ${ELASTIC_USER}
  password: ${ELASTIC_PASSWORD}
  xpack.enabled: true


- module: docker
  metricsets:
    - "container"
    - "cpu"
    - "diskio"
    - "healthcheck"
    - "info"
    #- "image"
    - "memory"
    - "network"
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
  enabled: true


processors:
  - add_host_metadata: ~
  - add_docker_metadata: ~


output.elasticsearch:
  hosts: ${ELASTIC_HOSTS}
  username: ${ELASTIC_USER}
  password: ${ELASTIC_PASSWORD}
  ssl:
    certificate: "certs/es01/es01.crt"
    certificate_authorities: "certs/ca/ca.crt"
    key: "certs/es01/es01.key"

 

4) Elasticsearch 인증서 구성

개발

# Certs 생성
## 어느 서버에서? (개발)EC2-DEV-ELASTIC-STACK-01
sudo su - svc
cd docker_compose
docker compose -f createcerts.yml up

# Certs 복붙
## 어느 서버에서? (개발)EC2-DEV-ELASTIC-STACK-01 
sudo su - root
cd /data001/elasticsearch/config/certs
cp -r * /data001/kibana/config/certs/
ls -al /data001/kibana/config/certs/
cp -r * /data001/metricbeat/config/certs/
ls -al /data001/metricbeat/config/certs/

 

운영

# Certs 생성
## 어느 서버에서? (운영)EC2-PROD-ELASTICSEARCH-01
sudo su - svc
cd docker_compose
docker compose -f createcerts.yml up


# Certs 복붙
## 어느 서버에서? (운영)EC2-PROD-ELASTICSEARCH-01
sudo su - root
cd /data001/elasticsearch/config
sudo ls -al certs
sudo tar -cvf certs-$(date +\%Y\%m\%d).tar certs/
sudo mv certs-$(date +\%Y\%m\%d).tar /home/ec2-user/
### tar 파일을 나머지 운영 서버의 /home/ec2-user로 복사해주기(필자는 Mobaxterm의 SFTP 툴 사용함)
### 복사 완료했으면 EC2-PROD-ELASTICSEARCH-01 서버의 /home/ec2-user 경로에 있는 tar 제거
sudo rm -f /home/ec2-user/certs-$(date +\%Y\%m\%d).tar
### MetricBeat 인증서 복사
TEMP_PATH=/data001/elasticsearch/config/certs
cp -r $TEMP_PATH/* /data001/metricbeat/config/certs/
ls -al /data001/metricbeat/config/certs/

## 어느 서버에서? (운영)EC2-PROD-ELASTICSEARCH-02 & EC2-PROD-ELASTICSEARCH-03
### (sudo 권한 말고 root로 수행하기)(디렉토리 권한 때문에 sudo mv *이 안먹음)
sudo su - root
TEMP_PATH=/data001/elasticsearch/config/certs
mv /home/ec2-user/certs-$(date +\%Y\%m\%d).tar $TEMP_PATH
tar -xvf $TEMP_PATH/certs-$(date +\%Y\%m\%d).tar -C $TEMP_PATH
rm -f $TEMP_PATH/certs-$(date +\%Y\%m\%d).tar
mv $TEMP_PATH/certs/* $TEMP_PATH
ls -al $TEMP_PATH
rm -rf $TEMP_PATH/certs
cp -r $TEMP_PATH/* /data001/metricbeat/config/certs/
ls -al /data001/metricbeat/config/certs/

## 어느 서버에서? (운영)EC2-PROD-KIBANA-01
### (sudo 권한 말고 root로 수행하기)(디렉토리 권한 때문에 sudo mv *이 안먹음)
sudo su - root
TEMP_PATH=/data001/kibana/config/certs
mv /home/ec2-user/certs-$(date +\%Y\%m\%d).tar $TEMP_PATH
tar -xvf $TEMP_PATH/certs-$(date +\%Y\%m\%d).tar -C $TEMP_PATH
rm -f $TEMP_PATH/certs-$(date +\%Y\%m\%d).tar
mv $TEMP_PATH/certs/* $TEMP_PATH
ls -al $TEMP_PATH
rm -rf $TEMP_PATH/certs

5) Docker container 배포

개발

## 어느 서버에서? (개발)EC2-DEV-ELASTIC-STACK-01 
sudo su - svc
cd ~/docker_compose

docker compose -f elasticsearch01.yml up -d
docker compose -f setup.yml up
docker compose -f kibana.yml up -d
docker compose -f metricbeat01.yml up -d

 

운영

## 어느 서버에서? (운영)운영 모든 서버에서
sudo su - svc
cd ~/docker_compose

## 어느 서버에서? (운영)EC2-PROD-ELASTICSEARCH-01
docker compose -f elasticsearch01.yml up -d
docker compose -f setup.yml up

## 어느 서버에서? (운영)EC2-PROD-ELASTICSEARCH-02
docker compose -f elasticsearch02.yml up -d

## 어느 서버에서? (운영)EC2-PROD-ELASTICSEARCH-03
docker compose -f elasticsearch03.yml up -d

## 어느 서버에서? (운영)EC2-PROD-KIBANA-01
docker compose -f kibana.yml up -d

## 어느 서버에서? (운영)EC2-PROD-ELASTICSEARCH-01
docker compose -f metricbeat01.yml up -d

## 어느 서버에서? (운영)EC2-PROD-ELASTICSEARCH-02
docker compose -f metricbeat02.yml up -d

## 어느 서버에서? (운영)EC2-PROD-ELASTICSEARCH-03
docker compose -f metricbeat03.yml up -d

6. 플러그인 설치

1) Nori 플러그인 설치

필요한 경우에만 설치 진행한다. 설치 방법은 아래와 같이 컨테이너 내부에 설치 명령어를 실행한다.
이렇게 설치한 경우 Docker container가 재생성되는 경우 사라지니 다시 설치해줘야 한다.
해당 플러그인이 설치된 이미지를 생성하여 사용하는 것을 권장한다.
또는 아직 테스트는 안 해봤는데 플러그인이 설치되는 경로를 마운트하여 컨테이너가 재생성되더라도 삭제되지 않도록 구성해 보자.

https://www.elastic.co/docs/reference/elasticsearch/plugins/analysis-nori

docker exec -it <Elasticsearch_컨테이너_이름> bin/elasticsearch-plugin install analysis-nori
docker exec -it <Elasticsearch_컨테이너_이름> bin/elasticsearch-plugin list
docker restart <Elasticsearch_컨테이너_이름>

7. 도메인 등록

.env에서 작성한 domain으로 Kibana에 접근할 수 있도록 도메인 등록

본 아키텍처에서는 생략했지만 EC2 앞단에 ALB를 생성하고 ACM에 인증서 등록해서 ALB에서 HTTPS 사용할 수 있도록 구성 (사용자-->인터넷-->ALB-->EC2)

만약 ALB을 앞단에 두지 않는다면 보안을 위해 Kibana를 http에서 https 구성으로 변경 필요

본 게시글에서는 Kibana http 구성 사용

8. Kibana Web Console에서 설정

Kibana에 로그인하여 아래 설정들을 셋팅한다.

superuser 권한을 가진 elastic User가 있으며, Password는 .env에서 설정한 KIBANA_PASSWORD 또는 ELASTIC_PASSWORD를 사용한다.

(지금까지 동일한 값으로 배포했어서 정확히 KIBANA_PASSWORD인지 ELASTIC_PASSWORD인지 잘모르겠다... KIBANA_SYSTEM이란 User가 있는데 그 User로도 동일 PASSWORD 인증은 되는 것 같아서 아마 elastic User는 ELASTIC_PASSWORD를 사용하는 것으로 추측)

1) Metricbeat

🔹 collection.enabled:false

Kibana DevTools에서 아래와 같이 설정해 준다.

본 설정은 7.16 버전부터 Deprecated 되었고 공식 문서에서 Metricbeat 사용을 권장하고 있다.

때문에 Metricbeat와 중복 수집을 방지하기 위해 false로 세팅해 준다.
근데 참고로 기본값이 false이긴 하다.

PUT _cluster/settings
{
    "persistent": {
        "xpack.monitoring.collection.enabled": false
    }
}

 

🔹 Metricbeat 배포 후 보관 주기 설정

Kibana Web Console > Stack Management > Data > Index Lifecycle Policies > metricbeat > Hot Phase에서 휴지통 아이콘 클릭 > Delete Phase가 활성화되면 5일로 설정하고 저장.
-https://www.elastic.co/guide/en/elasticsearch/reference/8.17/getting-started-index-lifecycle-management.html#ilm-gs-create-policy

 

🔹 Metircbeat 생성 시 함께 생성되는 것들()

 

 

2) 인덱스 템플릿

아래와 같이 인덱스 템플릿을 설정하면 인덱스 패턴에 일치하는 인덱스가 생성될 때 인덱스 템플릿의 구성을 따른다.
priority의 숫자가 클수록 우선순위가 높다.

(주의) 인덱스 패턴을 *로 설정할 경우 다른 인덱스 템플릿과 충돌하는 현상이 발생했다. 결국 권장 방법은 * 대신 인덱스 패턴의 범위를 좁혀서 사용하라고 한다. Elastisearch 최초 생성 후 인덱스 패턴을 *로 템플릿을 생성하면 생성이 되긴 하는데 대신 Metricbeat 배포가 안됐다. Metricbeat 관련 인덱스 배포 시 템플릿이 충돌해서 발생한 문제였다. 아래 예제와 같이 인덱스 패턴의 범위를 좁혀보자.

# 개발
PUT /_index_template/custom-all-number-of-shards-template
{
  "index_patterns": ["category-a-*"], 
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  },
  "priority": 1
}

# 운영
PUT /_index_template/custom-all-number-of-shards-template
{
  "index_patterns": ["category-a-*"], 
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 2
    }
  },
  "priority": 1
}

3) Kibana Time Zone

Kibana Web Console > Stack Management > Advanced Settings > Time Zone에서 Asia/Seoul로 설정하고 저장한다.

4) Elasticsearch Snapshot API

Elasticsearch에서 제공하는 백업 기능AWS 환경에서 EC2 여러대로 구성했을 때, 그리고 AWS Backup을 사용해서 특정 시간에 백업되도록 설정할 경우 정확히 같은 시간에 백업을 시작해서 정확히 같은 시간에 끝나지 않는다. 따라서 AWS Backup으로 백업한 AMI로 복구를 했을 경우 정합성이 어긋날 수 있다. 그래서 Elasticsearch에서 제고하는 Snapshot API를 사용해서 인덱스, 데이터스트림을 백업할 것을 권장한다.

Kibana Web Console > Stack Management > Snapshot and Restore

 

🔹 Repository

스냅샷을 저장할 공간. 본 글에서는 S3로 설정했다.

S3를 Repository로 설정할 경우 EC2에서 S3에 적정한 권한을 가지고 있어야한다.

필자는 아래 권한을 넣어줬다. (아마 GetObjectVersion은 필요 없을 것 같기도하다.)

DeleteObject / PutObject / GetObject / ListBucket / GetObjectVersion

 

🔹 Policy

여기부터 다시 작성하기

🔹 Snapshot

🔹 retention_schedule

PUT _cluster/settings
{
  "persistent" : {
    "slm.retention_schedule" : "0 0 16 * * ?"
  }
}

 

 

 

 

 

 

Elasticsearch 권한

Kibana encryption key

서버 간 통신

IAM Role Policy

s3

별첨. 이럴 수가

  1. Docker Container 로그가 저장되는 기본 위치는 /var/lib/docker/containers//-json.log이다. 로그가 쌓이는 경로를 손쉽게 변경할 수 있을 줄 알고 logs001 경로에 EBS를 별도로 마운트 해줬는데 생각보다 변경하기 까탈스러운 것 같다. 그래서 결국 기본 위치를 사용하기로 했다.
  2. Docker Container에 Mount 해줄 디렉토리의 소유권 및 접근 권한은 docker image의 Dockerfile 내 User를 고려해서 세팅해야 한다. Elasticsearch Docker Image의 경우 내부적으로 elasticsearch라는 User를 사용하는데 elasticsearch User는 1000번을 사용한다. 따라서 1000번으로 권한 설정을 해주었다. 참고로 Amazon linux 2023에서 1000번은 ec2-user이다.
  3. EC2의 Timezone과 Docker Continaer의 Timezone은 별개다.
  4. Index template의 Index Pattern을 *로 생성하면 다른 시스템에 문제가 발생할 수 있다.
  5. docker compose yml 작성 시 volume을 생성해서 사용하려고 했는데 volume을 생성하면 /var/lib/docker/volumes/ 하위에 생성됨. 특정 경로로 지정도 가능하다고는 하는데 docker compose down -v를 실수로 실행하면 볼륨도 함께 삭제돼서 굳이 volume 생성 안 하고 Bind Mount 방식을 사용함. 그래서 그냥 절대 경로를 사용해서 컨테이너에 마운트 해줌.

별첨. 복구 시 주의 사항

  1. ec2로 복구 시 hostname 리셋
  2. ec2 private ip 변경 시 hosts 파일 반영 필요
  3. EBS 자동 마운트되는지 확인 필요
반응형

티스토리툴바