# 배포를 위한 설정

- QKS 인벤토리 작성
- QKS 상세옵션 설정

### QKS 인벤토리 작성

인벤토리 파일 위치는 `inventory/qks/hosts`이며, 아래의 샘플을 참고합니다.

```ini
# 모든 호스트 목록
#
# 형식: <hostname> ansible_host=<IP> ansible_user=<USER> private_ip=<IP>
#
# <hostname>:
# 배포 과정에서 지정한 이름이 호스트 이름으로 설정됩니다.
# 고객이 내부 정책에 맞게 제공한 호스트 이름이 있다면, 해당 이름으로 대체합니다.
#
# ansible_host:
# Control 에서 SSH로 연결할 Managed 노드의 주소를 설정합니다.
#
# ansible_user:
# Control 에서 SSH로 연결할 Managed 노드의 사용자를 설정합니다.
#
# private_ip:
# Kubernetes 노드간 통신할 내부 주소를 지정합니다.
#
# 참고: Ansible Control 노드와 Managed 노드가 동일한 네트워크에 있다면
#      ansible_host 와 private_ip 는 동일하게 지정할 수 있습니다.
#
qks-k8s-m01 ansible_host=13.125.0.10 ansible_user=quantum private_ip=172.31.0.10
qks-k8s-m02 ansible_host=13.125.0.11 ansible_user=quantum private_ip=172.31.0.11
qks-k8s-m03 ansible_host=13.125.0.12 ansible_user=quantum private_ip=172.31.0.12
qks-k8s-w01 ansible_host=13.125.0.13 ansible_user=quantum private_ip=172.31.0.13
qks-k8s-w02 ansible_host=13.125.0.14 ansible_user=quantum private_ip=172.31.0.14
qks-k8s-w03 ansible_host=13.125.0.15 ansible_user=quantum private_ip=172.31.0.15
qks-k8s-x01 ansible_host=13.125.0.16 ansible_user=quantum private_ip=172.31.0.16

# 수정 금지
[kube-cluster:children]
kube-master
kube-worker

# Kubernetes Master로 설정될 호스트 목록
#
# 최소 3개 이상을 나열해야하며, 정족수를 충족하기 위해 홀수로 지정합니다.
#
[kube-master]
qks-k8s-m01
qks-k8s-m02
qks-k8s-m03

# 수정 금지
[kube-worker:children]
qks-worker
qks-nvidia

# Kubernetes Worker로 설정될 호스트 목록
[qks-worker]
qks-k8s-w01
qks-k8s-w02
qks-k8s-w03

# NVIDIA GPU 하드웨어가 장착된 호스트 목록
#
# GPU 워크로드를 담당할 K8S Worker로 설정이 됩니다.
#
[qks-nvidia]
qks-k8s-w01
qks-k8s-w02
qks-k8s-w03

# 수정 금지
[qks-server:children]
qks-pkg-server
qks-nfs-server

# 운영체제 패키지 저장소로 설정될 호스트 목록
#
# 외부 네트워크와 단절된 환경 (Air-gapped) 에서 운영체제 패키기 저장소로 설정됩니다.
# 다수의 호스트를 지정하면 저장소가 HA로 구성됩니다.
#
[qks-pkg-server]
qks-k8s-w01

# NFS 서버로 설정될 호스트
#
# 하드웨어 스토리지가 없을 경우만 지정하며, 하드웨어 스토리지가 있다면 생략합니다.
# NFS의 특성상 HA구성이 불가하므로 반드시 하나의 호스트만을 지정합니다.
#
[qks-nfs-server]
qks-k8s-x01

# CEPH 스토리지로 설정될 호스트 목록
#
# 하드웨어 스토리지가 없을 경우만 지정하며, 하드웨어 스토리지가 있다면 생략합니다.
# 최소 3개 이상을 나열해야하며, 정족수를 충족하기 위해 홀수로 지정합니다.
#
[qks-ceph]
qks-k8s-w01
qks-k8s-w02
qks-k8s-w03
```

> `[섹션]`의 호스트 목록은 없을 수 있지만, `[섹션]` 자체를 삭제하면 안됩니다.


### QKS 상세옵션 설정

상세옵션 파일 위치는 `inventory/qks/group_vars/all/qks.yaml`입니다.

```yaml
################################################################################
# QKS Kubernetes Environment ################################ BEGIN ###
################################################################################

# K8S 로드밸런서: 도메인
ext_lb_fqdn: "{{ qks_load_balancer_fqdn }}"
# K8S 로드밸런서: 주소
ext_lb_addr: "{{ qks_load_balancer_addr }}"
# K8S 로드밸런서: 포트
ext_lb_port: 8443

# K8S 서비스 : 버전
kube_version: 1.21.4
# K8S 서비스: POD CIDR
kube_pod_cidr: 10.0.0.0/16
# K8S 서비스: SVC CIDR
kube_svc_cidr: 10.1.0.0/16
# K8S 서비스: PROXY MODE
kube_proxy_mode: iptables

# K8S Master 노드 Tatint 여부
kube_master_node_taint: true

# Kubernetes Data Directory
kube_data_dir_cri:     /data/cri         # default: /var/lib/docker
kube_data_dir_kubelet: /data/k8s/kubelet # default: /var/lib/kubelet
kube_data_dir_etcd:    /data/k8s/etcd    # default: /var/lib/etcd

# Kubernetes admin users
kube_admin_users:
  - "{{ ansible_user }}"
  - "root"
  - "ec2-user" # for redhat on AWS
  - "ubuntu"   # for ubuntu on AWS

###############################
# QKS Kubernetes CNI #
###############################

# K8S CNI: 컨테이너 네트워크 선택
kube_cni: calico
# K8S CNI: Calico 버전
cni_calico_version: 3.17.1
# K8S CNI: Calico MTU
cni_calico_mtu: 0
# K8S CNI: Calico iptable 백엔드
cni_calico_iptablesbackend: Auto
cni_calico_cidr_autodetection_method: default


###############################
# QKS Kubernetes CRI #
###############################

# K8S CRI: 컨테이너 런타임 종류
kube_cri: containerd
# K8S CRI: 컨테이너 런타임 버전
kube_cri_version:
# K8S CRI 옵션: Cgroup 드라이버 선택
cri_cgroup_driver: systemd
# K8S CRI 옵션: Insecure 레지스트리
cri_insecure_registries:
  - "0.0.0.0/0"

kube_default_domain: qks.io

################################################################################
######################################################################## END ###
################################################################################

# QKS Offline Materials
qks_offline_enabled: true
qks_offline_source: "{{ inventory_dir }}/../../data/offline"
qks_offline_target: /data/qks/offline

# QKS Offline Registry Mirror
qks_offline_image_mirror: true

# QKS Service Materials
qks_service_source: "{{ inventory_dir }}/../../data/service"
qks_service_target: /data/qks/service

# QKS Collector
qks_collector_target: /data/qks/collector
qks_collector_gathering_deployer: true
qks_collector_gathering_packages: true
qks_collector_gathering_charts: true
qks_collector_gathering_files: true
qks_collector_gathering_images: true
qks_collector_cleanup: true

# QKS Menifess Location
qks_manifests_location: /etc/kubernetes/qks

# QKS Certificates Location
qks_certificates_location: /etc/kubernetes/qks/qks-certificates
qks_certificates_bits: 2048
qks_certificates_expiration: 3650 # specify in days
qks_certificates_recreation: false

# QKS Namespace
qks_system_namespace: qks-system
qks_service_namespace: qks

# QKS Package Repository
qks_package_repository_enabled: true
qks_package_repository_path: /repo
qks_package_repository_port: 1234
qks_package_repository_online_handling: all_disable

# QKS K8S Tools
qks_tools_enabled: true
qks_tools_powerline_version: 1.21.0 # https://github.com/justjanne/powerline-go
qks_tools_kubectx_version: 0.9.4    # https://github.com/ahmetb/kubectx
qks_tools_kubens_version: 0.9.4     # https://github.com/ahmetb/kubectx
qks_tools_k9scli_version: 0.24.15   # https://k9scli.io/
qks_tools_kubestr_version: 0.4.17   # https://kubestr.io/

# QKS Helm
qks_helm_enabled: true
qks_helm_version: 3.6.1
qks_helm_max_history: 10
qks_helm_stable_repo: "https://charts.helm.sh/stable"

# QKS Load Balancer
qks_vip_manager_enabled: false
qks_vip_manager_version: 2.0.20
qks_vip_manager_nic: eth0
qks_load_balancer_enabled: true
qks_load_balancer_version: 2.2.5
qks_load_balancer_namespace: "{{ qks_system_namespace }}"
qks_load_balancer_stats: true
qks_load_balancer_stats_port: 8888
qks_load_balancer_stats_user: admin
qks_load_balancer_stats_pass: QKSk8s
qks_load_balancer_vip: xxx.xxx.xxx.xxx
qks_load_balancer_fqdn: k8s.qks.io
qks_load_balancer_addr: "{% if qks_vip_manager_enabled | bool %}{{ qks_load_balancer_vip }}{% else %}{{ hostvars[groups['kube-master'][0]]['private_ip'] }}{% endif %}"
qks_load_balancer_config_location: "/etc/qks"

# QKS Metrics Server
qks_metrics_server_enabled: true
qks_metrics_server_release: qks-metrics-server
qks_metrics_server_version: 2.11.4
qks_metrics_server_namespace: "{{ qks_system_namespace }}"
qks_metrics_server_replicas: 2

# QKS Ingress Controller
qks_ingress_controller_enabled: true
qks_ingress_controller_release: qks-ingress
qks_ingress_controller_version: 3.29.0
qks_ingress_controller_namespace: "{{ qks_system_namespace }}"
qks_ingress_controller_replicas: 2
qks_ingress_controller_tlssecret: tls.qks.io
qks_ingress_controller_nodeport_insecure: 30080
qks_ingress_controller_nodeport_secure: 30443
qks_ingress_backend_enabled: true
qks_ingress_backend_replicas: 1

# QKS NFS Server
qks_nfs_server_enabled: true
qks_nfs_server_export_path: /nfs
qks_nfs_server_export_opts: "*(rw,sync,no_root_squash,fsid=0,no_subtree_check)"

# QKS Rook Ceph
qks_rook_ceph_enabled: true
qks_rook_ceph_release: qks-rook-ceph
qks_rook_ceph_version: 1.5.9
qks_rook_ceph_namespace: rook-ceph
qks_rook_ceph_hostnetwork: false
qks_rook_ceph_node_taint: false
qks_rook_ceph_admin_fqdn: ceph.qks.io
qks_rook_ceph_admin_pass: QKSk8s

qks_rook_ceph_image_version: 15.2.9
qks_rook_ceph_monitor_count: 3

qks_rook_ceph_storage_devices:
  - { host: "{{ groups['qks-ceph'][0] }}", device: nvme1n1, class: hdd }
  - { host: "{{ groups['qks-ceph'][1] }}", device: nvme1n1, class: hdd }
  - { host: "{{ groups['qks-ceph'][2] }}", device: nvme1n1, class: hdd }

# QKS Ceph Block Storage (rbd)
qks_rook_ceph_block_storage_enabled: true
qks_rook_ceph_block_storage_failuredomain: host
qks_rook_ceph_block_pool_name: qks-block
qks_rook_ceph_block_pool_type: r
qks_rook_ceph_block_pool_meta_deviceclass: hdd
qks_rook_ceph_block_pool_data_deviceclass: hdd
qks_rook_ceph_block_pool_replication_size: 3
qks_rook_ceph_block_storage_class_name: qks-ceph-block
qks_rook_ceph_block_storage_class_reclaimpolicy: Retain
qks_rook_ceph_block_stroage_class_fstype: ext4

# QKS Ceph Filesystem Storage (cephfs)
qks_rook_ceph_filesystem_storage_enabled: true
qks_rook_ceph_filesystem_storage_failuredomain: host
qks_rook_ceph_filesystem_pool_name: qks-cephfs
qks_rook_ceph_filesystem_pool_type: r
qks_rook_ceph_filesystem_pool_meta_deviceclass: hdd
qks_rook_ceph_filesystem_pool_data_deviceclass: hdd
qks_rook_ceph_filesystem_pool_replication_size: 3
qks_rook_ceph_filesystem_storage_class_name: qks-ceph-cephfs
qks_rook_ceph_filesystem_storage_class_reclaimpolicy: Retain
qks_rook_ceph_filesystem_mount_on_masters: false

# QKS Ceph Object Storage (s3)
qks_rook_ceph_object_storage_enabled: true
qks_rook_ceph_object_storage_failuredomain: host
qks_rook_ceph_object_storage_instances: 3
qks_rook_ceph_object_storage_fqdn: s3.qks.io
qks_rook_ceph_object_storage_port: 8080
qks_rook_ceph_object_pool_name: qks-object
qks_rook_ceph_object_pool_type: r
qks_rook_ceph_object_pool_meta_deviceclass: hdd
qks_rook_ceph_object_pool_data_deviceclass: hdd
qks_rook_ceph_object_pool_replication_size: 3

# QKS Ceph NFS Ganesha
qks_rook_ceph_ganesha_enabled: false
qks_rook_ceph_ganesha_failuredomain: host
qks_rook_ceph_ganesha_instances: 1
qks_rook_ceph_ganesha_pool_name: qks-ganesha
qks_rook_ceph_ganesha_pool_replication_size: 3
qks_rook_ceph_ganesha_pool_namespace: conf-ganesha
qks_rook_ceph_ganesha_export_path: ganesha

# QKS NFS Provisioner
qks_nfs_provisioner_enabled: true
qks_nfs_provisioner_release: qks-nfs-provisioner
qks_nfs_provisioner_version: 1.2.11
qks_nfs_provisioner_namespace: "{{ qks_system_namespace }}"
qks_nfs_provisioner_server: "{{ hostvars[groups['qks-nfs-server'][0]]['private_ip'] }}"
qks_nfs_provisioner_path: /nfs
qks_nfs_provisioner_replicas: 2
qks_nfs_provisioner_storageclass_name: qks-nfs
qks_nfs_provisioner_storageclass_reclaimpolicy: Retain
qks_nfs_provisioner_mount_on_masters: false

# QKS Harbor
qks_harbor_enabled: true
qks_harbor_release: qks-harbor
qks_harbor_version: 1.7.2
qks_harbor_namespace: qks-harbor
qks_harbor_fqdn_core: harbor.qks.io
qks_harbor_fqdn_notary: notary.qks.io
qks_harbor_pass: QKSk8s
qks_harbor_registry_storage_class: qks-ceph-cephfs
qks_harbor_registry_storage_size: 100Gi
qks_harbor_registry_storage_mode: ReadWriteMany
qks_harbor_chartmuseum_storage_class: qks-ceph-cephfs
qks_harbor_chartmuseum_storage_size: 1Gi
qks_harbor_chartmuseum_storage_mode: ReadWriteMany
qks_harbor_jobservice_storage_class: qks-ceph-cephfs
qks_harbor_jobservice_storage_size: 1Gi
qks_harbor_jobservice_storage_mode: ReadWriteMany
qks_harbor_database_storage_class: qks-ceph-cephfs
qks_harbor_database_storage_size: 10Gi
qks_harbor_database_storage_mode: ReadWriteMany
qks_harbor_redis_storage_class: qks-ceph-cephfs
qks_harbor_redis_storage_size: 10Gi
qks_harbor_redis_storage_mode: ReadWriteMany
qks_harbor_trivy_storage_class: qks-ceph-cephfs
qks_harbor_trivy_storage_size: 1Gi
qks_harbor_trivy_storage_mode: ReadWriteNany

# QKS Docker Registry
qks_registry_enabled: true
qks_registry_release: qks-registry
qks_registry_version: 1.9.6
qks_registry_namespace: "{{ qks_system_namespace }}"
qks_registry_replicas: 2
qks_registry_storage_class: qks-ceph-cephfs
qks_registry_storage_mode: ReadWriteMany
qks_registry_storage_size: 50Gi
qks_registry_fqdn: images.qks.io

# QKS Chartmuseum
qks_chartmuseum_enabled: false
qks_chartmuseum_release: qks-chartmuseum
qks_chartmuseum_version: 2.14.2
qks_chartmuseum_namespace: "{{ qks_system_namespace }}"
qks_chartmuseum_replicas: 2
qks_chartmuseum_storage_class: qks-ceph-cephfs
qks_chartmuseum_storage_mode: ReadWriteMany
qks_chartmuseum_storage_size: 1Gi
qks_chartmuseum_fqdn: charts.qks.io
qks_chartmuseum_name: qks
qks_chartmuseum_user: qks
qks_chartmuseum_pass: QKSk8s

# QKS Prometheus
qks_monitoring_enabled: true
qks_monitoring_release: qks-monitor
qks_monitoring_version: 18.0.1
qks_monitoring_namespace: qks-monitor
qks_monitoring_prometheus_fqdn: pc.qks.io
qks_monitoring_prometheus_retention: 4w
qks_monitoring_prometheus_storage_class: qks-ceph-cephfs
qks_monitoring_prometheus_storage_mode: ReadWriteMany
qks_monitoring_prometheus_storage_size: 50Gi
qks_monitoring_alertmanager_fqdn: pa.qks.io
qks_monitoring_alertmanager_retention: 120h
qks_monitoring_alertmanager_storage_class: qks-ceph-cephfs
qks_monitoring_alertmanager_storage_mode: ReadWriteMany
qks_monitoring_alertmanager_storage_size: 50Gi
qks_monitoring_grafana_fqdn: pm.qks.io
qks_monitoring_grafana_pass: QKSk8s
qks_monitoring_grafana_storage_class: qks-ceph-cephfs
qks_monitoring_grafana_storage_mode: ReadWriteMany
qks_monitoring_grafana_storage_size: 1Gi

# QKS GPU Accelerator
qks_accelerator_enabled: true
qks_accelerator_namespace: "{{ qks_system_namespace }}"
qks_accelerator_node_taint: false
qks_accelerator_nvidia_type: tesla
qks_accelerator_driver_version: 450.51.06
qks_accelerator_kernel_version:
  - "{{ ansible_kernel }}"
qks_accelerator_device_plugin_version: v0.9.0
qks_accelerator_device_metric_version: 2.1.4-2.3.1-ubuntu18.04


################################################################################
#                                                                              #
# QKS Solution Requirements                                           #
#                                                                              #
################################################################################

# QKS MariaDB
qks_mariadb_enabled: true
qks_mariadb_version: 7.10.4
qks_mariadb_release: qks-mariadb
qks_mariadb_namespace: "{{ qks_service_namespace }}"
qks_mariadb_admin_pass: QKSk8s
qks_mariadb_default_user_name: qks
qks_mariadb_default_user_pass: QKSk8s
qks_mariadb_default_user_database: qks
qks_mariadb_storage_class: qks-ceph-cephfs
qks_mariadb_storage_size: 50Gi
qks_mariadb_storage_mode: ReadWriteMany
qks_mariadb_slave_enabled: true
qks_mariadb_slave_replicas: 2
qks_mariadb_metrics_enabled: true

# QKS Keycloak
qks_keycloak_enabled: false
qks_keycloak_version: 14.0.1
qks_keycloak_release: qks-keycloak
qks_keycloak_namespace: "{{ qks_service_namespace }}"
qks_keycloak_fqdn: idp.qks.io
qks_keycloak_replicas: 2
qks_keycloak_discovery: DNS_PING # DNS_PING or KUBE_PING
qks_keycloak_admin_name: admin
qks_keycloak_admin_pass: QKSk8s
qks_keycloak_nodeport_insecure: 32080
qks_keycloak_nodeport_secure: 32443
qks_keycloak_db_vendor: mariadb
qks_keycloak_db_addr: "{{ qks_mariadb_release }}.{{ qks_mariadb_namespace }}"
qks_keycloak_db_port: 3306
qks_keycloak_db_name: keycloak
qks_keycloak_db_user: keycloak
qks_keycloak_db_pass: QKSk8s
qks_keycloak_theme: docker.io/tanggle/keycloak-themes:1.0
```

## 배포를 위한 파일

| 위치          | 용도                                                               |
| ------------- | ------------------------------------------------------------------ |
| data/deployer | 오프라인 배포시 필요한 파일들 (Python & Ansible)                   |
| data/offline  | 오프라인 배포시 필요한 파일들 (Images & Charts & Packages & Files) |
| data/service  | QKS 서비스를 위한 파일들 (Modeler & Pipeline)             |

## 배포를 위한 확인

```bash
ansible -i inventory/qks/hosts all -m "ping"
```
> 에러가 발생한다면 [배포를 위한 준비](#배포를-위한-준비)와 [배포를 위한 설정](#배포를-위한-설정)을 다시 확인합니다.

## 배포

```bash
cp -av qks.pem ~/
```
```bash
ansible-playbook -i inventory/qks/hosts qksk8s.yaml --flush-cache
```