kubernetes部署高可用Harbor

前言

本文Harbor高可用依照Harbor官网部署，主要思路如下，大家可以根据具体情况选择搭建。

部署Postgresql高可用集群。（本文选用Stolon进行管理，请查看文章《kubernetes下Stolon部署高可用Postgresql》）
部署redis高可用集群。（本文选用Helm对redis进行高可用部署，请查看文章《kubernetes部署高可用redis》，该文以整理好redis编排文件可直接使用）
部署Harbor高可用集群。（本文主要阐述Harbor的高可用部署，为《kubernetes搭建Harbor无坑及Harbor仓库同步》补充部分，请先行阅读）

一、Harbor部署前准备

本文仅说明高可用配置，其余部署请查看《kubernetes搭建Harbor无坑及Harbor仓库同步》

.安装方式

helm安装
直接使用博主整理好的编排文件安装（通过Helm生成）

1.helm安装

安装Helm请查看《kubernetes搭建Harbor无坑及Harbor仓库同步》，其中包含Helm安装。

1.1.下载 harbor-helm

git clone https://github.com/goharbor/harbor-helm.git

cd XXX/harbor-helm

1.2.修改value.yaml

database的Postgresql配置

database:

  # if external database is used, set "type" to "external"

  # and fill the connection informations in "external" section

  type: external

  internal:

    image:

      repository: goharbor/harbor-db

      tag: v1.8.2-dev

    # The initial superuser password for internal database

    password: "changeit"

    # resources:

    #  requests:

    #    memory: 256Mi

    #    cpu: 100m

    nodeSelector: {}

    tolerations: []

    affinity: {}

  external:

    host: "stolon-proxy-service" #管理postgresql的stolon的service，因为都在Pod中可相互访问

    port: "5432"

    username: "postgres"

    password: "password1"

    coreDatabase: "registry"

    clairDatabase: "clair"

    notaryServerDatabase: "notaryserver"

    notarySignerDatabase: "notarysigner"

    sslmode: "disable"

  ## Additional deployment annotations

  podAnnotations: {}

redis的配置

redis:

  # if external Redis is used, set "type" to "external"

  # and fill the connection informations in "external" section

  type: external

  internal:

    image:

      repository: goharbor/redis-photon

      tag: v1.8.2-dev

    # resources:

    #  requests:

    #    memory: 256Mi

    #    cpu: 100m

    nodeSelector: {}

    tolerations: []

    affinity: {}

  external:

    host: "10.8.4.133" #haproxy的地址通过haproxy管理redis集群

    port: "6379"

    # The "coreDatabaseIndex" must be "0" as the library Harbor

    # used doesn't support configuring it

    coreDatabaseIndex: "0"

    jobserviceDatabaseIndex: "1"

    registryDatabaseIndex: "2"

    chartmuseumDatabaseIndex: "3"

    password: ""

修改Harbor其他组件replicas(副本数)

# 例如nginx的副本数更改

nginx:

  image:

    repository: goharbor/nginx-photon

    tag: v1.8.2-dev

  replicas: 3

1.3.准备Harbor所需的registry、notarysigner、notaryserver、clair数据库，Harbor会自动在其中建表。

执行sql语句脚本，供stolon-init-database-job.yaml使用

cat <<EOF > ./postgresql.sh

#!/bin/bash

host="stolon-proxy-service"

user="postgres"

db="postgres"

export PGPASSWORD="password1"

args=(

        # force postgres to not use the local unix socket (test "external" connectibility)

        --host "$host"

        --username "$user"

        --dbname "$db"

        --quiet --no-align --tuples-only

)

if select="$(echo 'SELECT 1' | psql "${args[@]}")" && [ "$select" = '1' ]; then

   echo "====notaryserver==database==creating===="

   psql -h stolon-proxy-service -p 5432 -U postgres -f "/docker-entrypoint-initdb.d/initial-notaryserver.sql"

   echo "====notarysigner==database==creating===="

   psql -h stolon-proxy-service -p 5432 -U postgres -f "/docker-entrypoint-initdb.d/initial-notarysigner.sql"

   echo "====registry==database==creating===="

   psql -h stolon-proxy-service -p 5432 -U postgres -f "/docker-entrypoint-initdb.d/initial-registry.sql"

   echo "====clair==database==creating===="

   psql -h stolon-proxy-service -p 5432 -U postgres -f "/docker-entrypoint-initdb.d/initial-clair.sql"

   exit 0

fi

exit 1

EOF

创建registry数据库

cat <<EOF > ./initial-registry.sql

CREATE DATABASE registry ENCODING 'UTF8';

\c registry;

CREATE TABLE schema_migrations(version bigint not null primary key, dirty boolean not null);

EOF

创建notaryserver数据库

cat <<EOF > ./initial-notaryserver.sql

CREATE DATABASE notaryserver;

CREATE USER server;

alter user server with encrypted password 'password';

GRANT ALL PRIVILEGES ON DATABASE notaryserver TO server;

EOF

创建notarysigner数据库

cat <<EOF > ./initial-notarysigner.sql

CREATE DATABASE notarysigner;

CREATE USER signer;

alter user signer with encrypted password 'password';

GRANT ALL PRIVILEGES ON DATABASE notarysigner TO signer;

EOF

创建clair数据库

cat <<EOF > ./initial-clair.sql

CREATE DATABASE clair;

EOF

创建一个job的yaml（stolon-init-database-job.yaml），用于创建数据库，注意更改脚本的挂载位置，并复制脚本到各个节点或为node和yaml加上nodeselect标签，只在当前标签node下复制脚本

apiVersion: batch/v1

kind: Job

metadata:

  name: stolon-init-database-job

spec:

  template:

    spec:

      containers:

      - name: stolon-proxy

        image: sorintlab/stolon:master-pg10

        command:

          - "/bin/bash"

          - "/docker-entrypoint-initdb.d/postgresql.sh"

        volumeMounts:

        - mountPath: /docker-entrypoint-initdb.d

          name: database

      restartPolicy: OnFailure     #失败重启

      volumes:

        - name: database

          hostPath:

            path: /root/tmp/harbor/stolon/examples/kubernetes/sql

  activeDeadlineSeconds: 600   #10分钟没有complete，不再重启并移除Pod

1.3.部署Postgresql、redis

按照《kubernetes下Stolon部署高可用Postgresql》部署Postgresql，注意加入stolon-init-database-job.yaml。
按照《kubernetes部署高可用redis》部署redis，之后用haproxy管理redis集群（不可直接使用redis的service暴露，service会访问到slave节点，redis副本是只读不可写的，在harbor中会有报错）
部署haproxy
1. 安装haproxy
  
  yum -y install haproxy
  
  cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg-back
  
  vim /etc/haproxy/haproxy.cfg
2. 加入配置

defaults REDIS

mode tcp

timeout connect 1m

timeout server 6m

timeout client 6m

frontend ft_redis

 bind 0.0.0.0:6379 name redis

 default_backend bk_redis

backend bk_redis

 option tcp-check

 tcp-check connect

 tcp-check send PING\r\n

 tcp-check expect string +PONG

 tcp-check send info\ replication\r\n

 tcp-check expect string role:master

 tcp-check send QUIT\r\n

 tcp-check expect string +OK

 server R1 redis-0.redis-headless.default.svc.cluster.local:6379 check inter 1s

 server R2 redis-1.redis-headless.default.svc.cluster.local:6379 check inter 1s

 server R3 redis-2.redis-headless.default.svc.cluster.local:6379 check inter 1s

listen admin_stats

        stats   enable

        bind    *:9090

        mode    http

        option  httplog

        log     global

        maxconn 10

        stats   refresh 30s

        stats   uri /admin

        stats   realm haproxy

        stats   auth admin:admin

        stats   hide-version

        stats   admin if TRUE

systemctl start haproxy && systemctl enable haproxy && systemctl status haproxy

访问 harbor节点Ip:9090/admin如图所示，便成功

kubernetes部署高可用Harbor

⚠️k8s master节点高可用可阅读《haproxy+keepalive实现master集群高可用》

1.4.部署Harbor

安装harbor并将日志写入文件，可编辑文件保留.yaml编排文件，以便以后使用

helm install . --debug --name hub |sed 'w harbor.yaml'

或执行以下命令，编排chart不执行，作用生成编排文件，删除多余部分，进行使用

helm install . --debug --dry-run --name hub |sed 'w harbor.yaml'

2.通过整理好的编排文件执行

链接:https://pan.baidu.com/s/1cr1fnWGHc-70HAxx1YH4kg 密码:21a8

直接使用这个编排文件可能会有问题，最好勤劳以下使用helm跑，也可避免更改配置遗漏或错误的问题，适用用于实验，如若搭建请注意修改Volum、requestsource等Pod设置

秒客网