实现方式

复制同步
多harbor实例共享后端存储

主从复制同步

harbor官方默认提供主从复制的方案来解决镜像同步问题，通过复制的方式，我们可以实时将测试环境harbor仓库的镜像同步到生产环境harbor，类似于如下流程：

在实际生产运维的中，往往需要把镜像发布到几十或上百台集群节点上。这时，单个Registry已经无法满足大量节点的下载需求，因此要配置多个Registry实例做负载均衡。手工维护多个Registry实例上的镜像，将是十分繁琐的事情。Harbor可以支持一主多从的镜像发布模式，可以解决大规模镜像发布的难题：

只要往一台Registry上发布，镜像就像“仙女散花”般地同步到多个Registry中，高效可靠。
如果是地域分布较广的集群，还可以采用层次型发布方式，如从集团总部同步到省公司，从省公司再同步到市公司：

然而单靠主从同步，仍然解决不了harbor主节点的单点问题。

双主复制同步

所谓的双主复制其实就是复用主从同步实现两个harbor节点之间的双向同步，来保证数据的一致性，然后在两台harbor前端顶一个负载均衡器将进来的请求分流到不同的实例中去，只要有一个实例中有了新的镜像，就是自动的同步复制到另外的的实例中去，这样实现了负载均衡，也避免了单点故障，在一定程度上实现了Harbor的高可用性：

这个方案有一个问题就是有可能两个Harbor实例中的数据不一致。假设如果一个实例A挂掉了，这个时候有新的镜像进来，那么新的镜像就会在另外一个实例B中，后面即使恢复了挂掉的A实例，Harbor实例B也不会自动去同步镜像，这样只能手动的先关掉Harbor实例B的复制策略，然后再开启复制策略，才能让实例B数据同步，让两个实例的数据一致。
在实际生产使用中，主从复制十分的不靠谱。
所以这里推荐使用下面要说的这种方案。

多harbor实例共享后端存储

共享后端存储算是一种比较标准的方案，就是多个Harbor实例共享同一个后端存储，任何一个实例持久化到存储的镜像，都可被其他实例中读取。通过前置LB进来的请求，可以分流到不同的实例中去处理，这样就实现了负载均衡，也避免了单点故障：

这个方案在实际生产环境中部署需要考虑三个问题：

1. 共享存储的选取，Harbor的后端存储目前支持AWS S3、Openstack Swift, Ceph等，在我们的实验环境里，就直接使用nfs

2. Session在不同的实例上共享，这个现在其实已经不是问题了，在最新的harbor中，默认session会存放在redis中，我们只需要将redis独立出来即可。可以通过redis sentinel或者redis cluster等方式来保证redis的可用性。在我们的实验环境里，仍然使用单台redis

3. Harbor多实例数据库问题，这个也只需要将harbor中的数据库拆出来独立部署即可。让多实例共用一个外部数据库，数据库的高可用也可以通过数据库的高可用方案保证。

环境说明

实验环境

配置说明

安装配置NFS

1、安装NFS
root@storage:~# apt install nfs-kernel-server nfs-common
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following additional packages will be installed:
keyutils libnfsidmap2 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libtirpc1 python python-minimal python2.7 python2.7-minimal rpcbind
Suggested packages:
watchdog python-doc python-tk python2.7-doc binutils binfmt-support
The following NEW packages will be installed:
keyutils libnfsidmap2 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libtirpc1 nfs-common nfs-kernel-server python python-minimal python2.7
python2.7-minimal rpcbind
0 upgraded, 13 newly installed, 0 to remove and 4 not upgraded.
Need to get 4,344 kB of archives.
After this operation, 18.4 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://mirrors.aliyun.com/ubuntu xenial/main amd64 libnfsidmap2 amd64 0.25-5 [32.2 kB]
Get:2 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libpython2.7-minimal amd64 2.7.12-1ubuntu0~16.04.4 [339 kB]
Get:3 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 python2.7-minimal amd64 2.7.12-1ubuntu0~16.04.4 [1,261 kB]
Get:4 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 python-minimal amd64 2.7.12-1~16.04 [28.1 kB]
Get:5 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libpython2.7-stdlib amd64 2.7.12-1ubuntu0~16.04.4 [1,880 kB]
Get:6 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 python2.7 amd64 2.7.12-1ubuntu0~16.04.4 [224 kB]
Get:7 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libpython-stdlib amd64 2.7.12-1~16.04 [7,768 B]
Get:8 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 python amd64 2.7.12-1~16.04 [137 kB]
Get:9 http://mirrors.aliyun.com/ubuntu xenial/main amd64 keyutils amd64 1.5.9-8ubuntu1 [47.1 kB]
Get:10 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libtirpc1 amd64 0.2.5-1ubuntu0.1 [75.4 kB]
Get:11 http://mirrors.aliyun.com/ubuntu xenial/main amd64 rpcbind amd64 0.2.3-0.2 [40.3 kB]
Get:12 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 nfs-common amd64 1:1.2.8-9ubuntu12.1 [184 kB]
Get:13 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 nfs-kernel-server amd64 1:1.2.8-9ubuntu12.1 [88.0 kB]
Fetched 4,344 kB in 0s (5,046 kB/s)
Selecting previously unselected package libnfsidmap2:amd64.
(Reading database … 60656 files and directories currently installed.)
Preparing to unpack …/libnfsidmap2_0.25-5_amd64.deb …
Unpacking libnfsidmap2:amd64 (0.25-5) …
Selecting previously unselected package libpython2.7-minimal:amd64.
Preparing to unpack …/libpython2.7-minimal_2.7.12-1ubuntu0~16.04.4_amd64.deb …
Unpacking libpython2.7-minimal:amd64 (2.7.12-1ubuntu0~16.04.4) …
Selecting previously unselected package python2.7-minimal.
Preparing to unpack …/python2.7-minimal_2.7.12-1ubuntu0~16.04.4_amd64.deb …
Unpacking python2.7-minimal (2.7.12-1ubuntu0~16.04.4) …
Selecting previously unselected package python-minimal.
Preparing to unpack …/python-minimal_2.7.12-1~16.04_amd64.deb …
Unpacking python-minimal (2.7.12-1~16.04) …
Selecting previously unselected package libpython2.7-stdlib:amd64.
Preparing to unpack …/libpython2.7-stdlib_2.7.12-1ubuntu0~16.04.4_amd64.deb …
Unpacking libpython2.7-stdlib:amd64 (2.7.12-1ubuntu0~16.04.4) …
Selecting previously unselected package python2.7.
Preparing to unpack …/python2.7_2.7.12-1ubuntu0~16.04.4_amd64.deb …
Unpacking python2.7 (2.7.12-1ubuntu0~16.04.4) …
Selecting previously unselected package libpython-stdlib:amd64.
Preparing to unpack …/libpython-stdlib_2.7.12-1~16.04_amd64.deb …
Unpacking libpython-stdlib:amd64 (2.7.12-1~16.04) …
Processing triggers for man-db (2.7.5-1) …
Processing triggers for mime-support (3.59ubuntu1) …
Setting up libpython2.7-minimal:amd64 (2.7.12-1ubuntu0~16.04.4) …
Setting up python2.7-minimal (2.7.12-1ubuntu0~16.04.4) …
Linking and byte-compiling packages for runtime python2.7…
Setting up python-minimal (2.7.12-1~16.04) …
Selecting previously unselected package python.
(Reading database … 61412 files and directories currently installed.)
Preparing to unpack …/python_2.7.12-1~16.04_amd64.deb …
Unpacking python (2.7.12-1~16.04) …
Selecting previously unselected package keyutils.
Preparing to unpack …/keyutils_1.5.9-8ubuntu1_amd64.deb …
Unpacking keyutils (1.5.9-8ubuntu1) …
Selecting previously unselected package libtirpc1:amd64.
Preparing to unpack …/libtirpc1_0.2.5-1ubuntu0.1_amd64.deb …
Unpacking libtirpc1:amd64 (0.2.5-1ubuntu0.1) …
Selecting previously unselected package rpcbind.
Preparing to unpack …/rpcbind_0.2.3-0.2_amd64.deb …
Unpacking rpcbind (0.2.3-0.2) …
Selecting previously unselected package nfs-common.
Preparing to unpack …/nfs-common_1%3a1.2.8-9ubuntu12.1_amd64.deb …
Unpacking nfs-common (1:1.2.8-9ubuntu12.1) …
Selecting previously unselected package nfs-kernel-server.
Preparing to unpack …/nfs-kernel-server_1%3a1.2.8-9ubuntu12.1_amd64.deb …
Unpacking nfs-kernel-server (1:1.2.8-9ubuntu12.1) …
Processing triggers for man-db (2.7.5-1) …
Processing triggers for libc-bin (2.23-0ubuntu10) …
Processing triggers for systemd (229-4ubuntu21.15) …
Processing triggers for ureadahead (0.100.0-19) …
Setting up libnfsidmap2:amd64 (0.25-5) …
Setting up libpython2.7-stdlib:amd64 (2.7.12-1ubuntu0~16.04.4) …
Setting up python2.7 (2.7.12-1ubuntu0~16.04.4) …
Setting up libpython-stdlib:amd64 (2.7.12-1~16.04) …
Setting up python (2.7.12-1~16.04) …
Setting up keyutils (1.5.9-8ubuntu1) …
Setting up libtirpc1:amd64 (0.2.5-1ubuntu0.1) …
Setting up rpcbind (0.2.3-0.2) …
Setting up nfs-common (1:1.2.8-9ubuntu12.1) …

Creating config file /etc/idmapd.conf with new version

Creating config file /etc/default/nfs-common with new version
Adding system user statd' (UID 111) ...<br />Adding new userstatd’ (UID 111) with group nogroup' ...<br />Not creating home directory/var/lib/nfs’.
nfs-utils.service is a disabled or a static unit, not starting it.
Setting up nfs-kernel-server (1:1.2.8-9ubuntu12.1) …

Creating config file /etc/exports with new version

Creating config file /etc/default/nfs-kernel-server with new version
Processing triggers for libc-bin (2.23-0ubuntu10) …
Processing triggers for systemd (229-4ubuntu21.15) …
Processing triggers for ureadahead (0.100.0-19) …

2、配置NFS
root@storage:~# mkdir /data
root@storage:~# vim /etc/exports
root@storage:~/redis-mysql# cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)
#
/data (rw,no_root_squash)
root@storage:~# chmod 777 -R /data/
root@storage:~# systemctl restart nfs-server.service
root@storage:~# showmount -e
Export list for storage.dc.com:
/data

安装redis和mysql

这里我们就直接通过docker安装，docker-compose.yml文件内容如下：
root@storage:~/redis-mysql# cat docker-compose.yml
version: ‘3’
services:
mysql-server:
hostname: mysql-server
container_name: mysql-server
image: mysql:5.7
network_mode: host
volumes:
- /mysql57/data:/var/lib/mysql
command: —character-set-server=utf8
environment:
MYSQL_ROOT_PASSWORD: 123456
redis:
hostname: redis-server
container_name: redis-server
image: redis:3
network_mode: host

启动运行

root@storage:~/redis-mysql# docker-compose up -d
Pulling mysql-server (mysql:5.7)…
5.7: Pulling from library/mysql
177e7ef0df69: Pull complete
cac25352c4c8: Pull complete
8585afabb40a: Pull complete
1e4af4996053: Pull complete
c326522894da: Pull complete
9020d6b6b171: Pull complete
55eb37ec6e5f: Pull complete
1a9d2f77e0e7: Pull complete
d7e648ad64aa: Pull complete
4120d828ea6b: Pull complete
3b39dc5451af: Pull complete
Pulling redis (redis:3)…
3: Pulling from library/redis
f17d81b4b692: Pull complete
b32474098757: Pull complete
8980cabe8bc2: Pull complete
58af19693e78: Pull complete
a977782cf22d: Pull complete
9c1e268980b7: Pull complete
Creating redis-server … done
Creating mysql-server … done

root@storage:~/redis-mysql# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e6533520c245 mysql:5.7 “docker-entrypoint.s…” 28 minutes ago Up 28 minutes mysql-server
a26c846816b0 redis:3 “docker-entrypoint.s…” 28 minutes ago Up 28 minutes redis-server

导入registry数据库

配置好了mysql以后，还需要往mysql数据库中导入harbor registry库。安装了一个单机版harbor，启动了一个mysql，里面有一个registry数据库，直接导出来，然后再导入到新数据库中：
# 导出数据库：

docker exec -it harbor_db /bin/bash
mysqldump -uroot -p —databases registry > registry.dump

# 在宿主机上将registry.dump复制出来
docker cp harbor_db:/registry.dump ./

# 将宿主机上的registry.dump复制到独立的mysql容器中

docker cp ./registry.dump :/registry.dump

# 在独立的mysql容器将将registry数据库导入

docker exec -it /bin/bash

mysql -uroot -p

mysql> source /registry.dump

配置harbor

挂载NFS目录
root@harbor01:~# showmount -e 172.16.251.133
Export list for 172.16.251.133:
/data *
root@harbor01:~# mkdir /data
root@harbor01:~# mount 172.16.251.133:/data /data
root@harbor01:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 0 7.9G 0% /dev
tmpfs 1.6G 8.9M 1.6G 1% /run
/dev/sda3 42G 2.0G 38G 5% /
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda1 464M 59M 377M 14% /boot
tmpfs 1.6G 0 1.6G 0% /run/user/0
172.16.251.133:/data 18G 2.6G 15G 16% /data

生成证书

root@harbor01:~# openssl genrsa -des3 -out server.key 1024
Generating RSA private key, 1024 bit long modulus
…………++++++
………………++++++
e is 65537 (0x10001)
Enter pass phrase for server.key:
Verifying - Enter pass phrase for server.key:

root@harbor01:~# openssl rsa -in server.key -out server.key
Enter pass phrase for server.key:

root@harbor01:~# openssl req -new -key server.key -out server.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter ‘.’, the field will be left blank.
——-
Country Name (2 letter code) [AU]:CN
State or Province Name (full name) [Some-State]:Shanghai
Locality Name (eg, city) []:Pudong
Organization Name (eg, company) [Internet Widgits Pty Ltd]:BL
Organizational Unit Name (eg, section) []:IT
Common Name (e.g. server FQDN or YOUR name) []:harbor.dc.com
Email Address []:root@harbor.dc.com

Please enter the following ‘extra’ attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

root@harbor01:~# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
Signature ok
subject=/C=CN/ST=Shanghai/L=Pudong/O=BL/OU=IT/CN=harbor.dc.com/emailAddress=root@harbor.dc.com
Getting Private key

配置harbor.cfg

解压harbor-offline-installer-v1.5.2.tgz进入harbor目录
root@harbor01:~/harbor# cat harbor.cfg |grep -v ‘^$’ |grep -v ‘^#’
_version = 1.5.0
hostname = harbor.dc.com
ui_url_protocol = https
max_job_workers = 50
customize_crt = on
ssl_cert = /data/certs/server.crt
ssl_cert_key = /data/certs/server.key
secretkey_path = /data
admiral_url = NA
log_rotate_count = 50
log_rotate_size = 200M
http_proxy =
https_proxy =
no_proxy = 127.0.0.1,localhost,ui
email_identity =
email_server = smtp.mydomain.com
email_server_port = 25
email_username = sample_admin@mydomain.com
email_password = abc
email_from = admin sample_admin@mydomain.com
email_ssl = false
email_insecure = false
harbor_admin_password = Harbor12345
auth_mode = db_auth
ldap_url = ldaps://ldap.mydomain.com
ldap_basedn = ou=people,dc=mydomain,dc=com
ldap_uid = uid
ldap_scope = 2
ldap_timeout = 5
ldap_verify_cert = true
ldap_group_basedn = ou=group,dc=mydomain,dc=com
ldap_group_filter = objectclass=group
ldap_group_gid = cn
ldap_group_scope = 2
self_registration = on
token_expiration = 30
project_creation_restriction = everyone
db_host = 172.16.251.133
db_password = 123456
db_port = 3306
db_user = root
redis_url = 172.16.251.133:6379
clair_db_host = postgres
clair_db_password = password
clair_db_port = 5432
clair_db_username = postgres
clair_db = postgres
uaa_endpoint = uaa.mydomain.org
uaa_clientid = id
uaa_clientsecret = secret
uaa_verify_cert = true
uaa_ca_cert = /path/to/ca.pem
registry_storage_provider_name = filesystem
registry_storage_provider_config =

部署harbor

root@harbor01:~/harbor# cp ha/docker-compose.yml .
root@harbor01:~/harbor# ./prepare
Clearing the configuration file: ./common/config/jobservice/config.yml
Clearing the configuration file: ./common/config/jobservice/env
Clearing the configuration file: ./common/config/db/env
Clearing the configuration file: ./common/config/adminserver/env
Clearing the configuration file: ./common/config/ui/app.conf
Clearing the configuration file: ./common/config/ui/env
Clearing the configuration file: ./common/config/ui/private_key.pem
Clearing the configuration file: ./common/config/registry/root.crt
Clearing the configuration file: ./common/config/registry/config.yml
Clearing the configuration file: ./common/config/log/logrotate.conf
Clearing the configuration file: ./common/config/nginx/cert/server.key
Clearing the configuration file: ./common/config/nginx/cert/server.crt
Clearing the configuration file: ./common/config/nginx/nginx.conf
loaded secret from file: /data/secretkey
Generated configuration file: ./common/config/nginx/nginx.conf
Generated configuration file: ./common/config/adminserver/env
Generated configuration file: ./common/config/ui/env
Generated configuration file: ./common/config/registry/config.yml
Generated configuration file: ./common/config/db/env
Generated configuration file: ./common/config/jobservice/env
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/log/logrotate.conf
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/ui/app.conf
Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.

root@harbor01:~/harbor# ./install.sh

[Step 0]: checking installation environment …

Note: docker version: 18.09.0

Note: docker-compose version: 1.23.2

[Step 1]: loading Harbor images …
Loaded image: vmware/registry-photon:v2.6.2-v1.5.2
Loaded image: vmware/photon:1.0
b9a9313d1f80: Loading layer [==================================================>] 457.3MB/457.3MB
a17318dff4d5: Loading layer [==================================================>] 9.216kB/9.216kB
11b0dc760d85: Loading layer [==================================================>] 9.216kB/9.216kB
ab253bfa0789: Loading layer [==================================================>] 7.68kB/7.68kB
7c41921b687e: Loading layer [==================================================>] 1.536kB/1.536kB
Loaded image: vmware/mariadb-photon:v1.5.2
Loaded image: vmware/harbor-log:v1.5.2
6c8870489188: Loading layer [==================================================>] 61.24MB/61.24MB
Loaded image: vmware/nginx-photon:v1.5.2
d6e1f5663df8: Loading layer [==================================================>] 79.23MB/79.23MB
1e7414733b92: Loading layer [==================================================>] 10.95MB/10.95MB
740974ce05ad: Loading layer [==================================================>] 17.3MB/17.3MB
8e5664ede767: Loading layer [==================================================>] 15.87kB/15.87kB
86e94116f829: Loading layer [==================================================>] 3.072kB/3.072kB
173729674517: Loading layer [==================================================>] 28.24MB/28.24MB
Loaded image: vmware/notary-signer-photon:v0.5.1-v1.5.2
22c53afb5f7e: Loading layer [==================================================>] 151MB/151MB
2b4cfcba43e3: Loading layer [==================================================>] 6.656kB/6.656kB
11d44832002d: Loading layer [==================================================>] 2.048kB/2.048kB
f805651773a5: Loading layer [==================================================>] 7.68kB/7.68kB
Loaded image: vmware/postgresql-photon:v1.5.2
b60d71996f91: Loading layer [==================================================>] 11.78kB/11.78kB
9f7d20f3523b: Loading layer [==================================================>] 2.56kB/2.56kB
1928f43498e2: Loading layer [==================================================>] 3.072kB/3.072kB
Loaded image: vmware/harbor-db:v1.5.2
74bdb194304b: Loading layer [==================================================>] 79.23MB/79.23MB
97251db4b4f7: Loading layer [==================================================>] 20.91MB/20.91MB
7c3dce375b3a: Loading layer [==================================================>] 20.91MB/20.91MB
Loaded image: vmware/harbor-jobservice:v1.5.2
3e7aac7aae19: Loading layer [==================================================>] 212.2MB/212.2MB
8e7d884dc74c: Loading layer [==================================================>] 10.93MB/10.93MB
12ed92f1f090: Loading layer [==================================================>] 2.048kB/2.048kB
81a629ba5751: Loading layer [==================================================>] 48.13kB/48.13kB
2171b5a262b0: Loading layer [==================================================>] 10.97MB/10.97MB
Loaded image: vmware/clair-photon:v2.0.4-v1.5.2
b5630b8d91ab: Loading layer [==================================================>] 79.23MB/79.23MB
89643b845018: Loading layer [==================================================>] 15.37MB/15.37MB
7b6fa7dbcef0: Loading layer [==================================================>] 15.37MB/15.37MB
Loaded image: vmware/harbor-adminserver:v1.5.2
bae5fd08868a: Loading layer [==================================================>] 79.24MB/79.24MB
ca9e737c9888: Loading layer [==================================================>] 24.41MB/24.41MB
5630a0e582d3: Loading layer [==================================================>] 7.168kB/7.168kB
68eedf2397cc: Loading layer [==================================================>] 10.56MB/10.56MB
74ac61aa9a2b: Loading layer [==================================================>] 24.4MB/24.4MB
Loaded image: vmware/harbor-ui:v1.5.2
1b5a1160526a: Loading layer [==================================================>] 136.2MB/136.2MB
24d9672d5b26: Loading layer [==================================================>] 3.072kB/3.072kB
fe4d52841f3d: Loading layer [==================================================>] 59.9kB/59.9kB
bffe30bd1941: Loading layer [==================================================>] 61.95kB/61.95kB
Loaded image: vmware/redis-photon:v1.5.2
dccbbe2d4eff: Loading layer [==================================================>] 12.16MB/12.16MB
d1291eaee2a3: Loading layer [==================================================>] 17.3MB/17.3MB
610d6a304e63: Loading layer [==================================================>] 15.87kB/15.87kB
2dabac4548fb: Loading layer [==================================================>] 3.072kB/3.072kB
46189122932e: Loading layer [==================================================>] 29.46MB/29.46MB
Loaded image: vmware/notary-server-photon:v0.5.1-v1.5.2
454c81edbd3b: Loading layer [==================================================>] 135.2MB/135.2MB
e99db1275091: Loading layer [==================================================>] 395.4MB/395.4MB
051e4ee23882: Loading layer [==================================================>] 9.216kB/9.216kB
6cca4437b6f6: Loading layer [==================================================>] 9.216kB/9.216kB
1d48fc08c8bc: Loading layer [==================================================>] 7.68kB/7.68kB
0419724fd942: Loading layer [==================================================>] 1.536kB/1.536kB
543c0c1ee18d: Loading layer [==================================================>] 655.2MB/655.2MB
4190aa7e89b8: Loading layer [==================================================>] 103.9kB/103.9kB
Loaded image: vmware/harbor-migrator:v1.5.0

[Step 2]: preparing environment …
Clearing the configuration file: ./common/config/jobservice/config.yml
Clearing the configuration file: ./common/config/jobservice/env
Clearing the configuration file: ./common/config/db/env
Clearing the configuration file: ./common/config/adminserver/env
Clearing the configuration file: ./common/config/ui/app.conf
Clearing the configuration file: ./common/config/ui/env
Clearing the configuration file: ./common/config/ui/private_key.pem
Clearing the configuration file: ./common/config/registry/root.crt
Clearing the configuration file: ./common/config/registry/config.yml
Clearing the configuration file: ./common/config/log/logrotate.conf
Clearing the configuration file: ./common/config/nginx/cert/server.key
Clearing the configuration file: ./common/config/nginx/cert/server.crt
Clearing the configuration file: ./common/config/nginx/nginx.conf
loaded secret from file: /data/secretkey
Generated configuration file: ./common/config/nginx/nginx.conf
Generated configuration file: ./common/config/adminserver/env
Generated configuration file: ./common/config/ui/env
Generated configuration file: ./common/config/registry/config.yml
Generated configuration file: ./common/config/db/env
Generated configuration file: ./common/config/jobservice/env
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/log/logrotate.conf
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/ui/app.conf
Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.

[Step 3]: checking existing instance of Harbor …

[Step 4]: starting Harbor …
Creating harbor-log … done
Creating registry … done
Creating harbor-adminserver … done
Creating harbor-ui … done
Creating nginx … done
Creating harbor-jobservice … done

✔ ——Harbor has been installed and started successfully.——

Now you should be able to visit the admin portal at https://harbor01.dc.com.
For more details, please visit https://github.com/vmware/harbor .

root@harbor01:~/harbor# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cdf3685c2d94 vmware/harbor-jobservice:v1.5.2 “/harbor/start.sh” 13 seconds ago Up 12 seconds harbor-jobservice
5de16bad4f2d vmware/nginx-photon:v1.5.2 “nginx -g ‘daemon of…” 13 seconds ago Up 12 seconds (health: starting) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp nginx
15406c90e365 vmware/harbor-ui:v1.5.2 “/harbor/start.sh” 14 seconds ago Up 13 seconds (health: starting) harbor-ui
9e7330ce8130 vmware/harbor-adminserver:v1.5.2 “/harbor/start.sh” 14 seconds ago Up 13 seconds (health: starting) harbor-adminserver
d344a7ae027e vmware/registry-photon:v2.6.2-v1.5.2 “/entrypoint.sh serv…” 14 seconds ago Up 13 seconds (health: starting) 5000/tcp registry
3c84cf5271f6 vmware/harbor-log:v1.5.2 “/bin/sh -c /usr/loc…” 15 seconds ago Up 14 seconds (health: starting) 127.0.0.1:1514->10514/tcp harbor-log
root@harbor01:~/harbor#

harbor测试

安装配置LVS

root@Userver05:~# apt install ipvsadm keepalived
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following additional packages will be installed:
libnl-route-3-200 libsensors4 libsnmp-base libsnmp30
Suggested packages:
heartbeat ldirectord lm-sensors snmp-mibs-downloader
The following NEW packages will be installed:
ipvsadm keepalived libnl-route-3-200 libsensors4 libsnmp-base libsnmp30
0 upgraded, 6 newly installed, 0 to remove and 9 not upgraded.
Need to get 1,429 kB of archives.
After this operation, 5,570 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libnl-route-3-200 amd64 3.2.27-1ubuntu0.16.04.1 [124 kB]
Get:2 http://mirrors.aliyun.com/ubuntu xenial/main amd64 libsensors4 amd64 1:3.4.0-2 [28.4 kB]
Get:3 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libsnmp-base all 5.7.3+dfsg-1ubuntu4.2 [224 kB]
Get:4 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libsnmp30 amd64 5.7.3+dfsg-1ubuntu4.2 [813 kB]
Get:5 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 keepalived amd64 1:1.2.24-1ubuntu0.16.04.1 [205 kB]
Get:6 http://mirrors.aliyun.com/ubuntu xenial/main amd64 ipvsadm amd64 1:1.28-3 [35.5 kB]
Fetched 1,429 kB in 0s (2,824 kB/s)
Selecting previously unselected package libnl-route-3-200:amd64.
(Reading database … 60370 files and directories currently installed.)
Preparing to unpack …/libnl-route-3-200_3.2.27-1ubuntu0.16.04.1_amd64.deb …
Unpacking libnl-route-3-200:amd64 (3.2.27-1ubuntu0.16.04.1) …
Selecting previously unselected package libsensors4:amd64.
Preparing to unpack …/libsensors4_1%3a3.4.0-2_amd64.deb …
Unpacking libsensors4:amd64 (1:3.4.0-2) …
Selecting previously unselected package libsnmp-base.
Preparing to unpack …/libsnmp-base_5.7.3+dfsg-1ubuntu4.2_all.deb …
Unpacking libsnmp-base (5.7.3+dfsg-1ubuntu4.2) …
Selecting previously unselected package libsnmp30:amd64.
Preparing to unpack …/libsnmp30_5.7.3+dfsg-1ubuntu4.2_amd64.deb …
Unpacking libsnmp30:amd64 (5.7.3+dfsg-1ubuntu4.2) …
Selecting previously unselected package keepalived.
Preparing to unpack …/keepalived_1%3a1.2.24-1ubuntu0.16.04.1_amd64.deb …
Unpacking keepalived (1:1.2.24-1ubuntu0.16.04.1) …
Selecting previously unselected package ipvsadm.
Preparing to unpack …/ipvsadm_1%3a1.28-3_amd64.deb …
Unpacking ipvsadm (1:1.28-3) …
Processing triggers for libc-bin (2.23-0ubuntu10) …
Processing triggers for man-db (2.7.5-1) …
Processing triggers for systemd (229-4ubuntu21.15) …
Processing triggers for ureadahead (0.100.0-19) …
Setting up libnl-route-3-200:amd64 (3.2.27-1ubuntu0.16.04.1) …
Setting up libsensors4:amd64 (1:3.4.0-2) …
Setting up libsnmp-base (5.7.3+dfsg-1ubuntu4.2) …
Setting up libsnmp30:amd64 (5.7.3+dfsg-1ubuntu4.2) …
Setting up keepalived (1:1.2.24-1ubuntu0.16.04.1) …
Setting up ipvsadm (1:1.28-3) …
Processing triggers for libc-bin (2.23-0ubuntu10) …
Processing triggers for systemd (229-4ubuntu21.15) …
Processing triggers for ureadahead (0.100.0-19) …

root@Userver05:~# cat /etc/keepalived/keepalived.conf
global_defs {
router_id haborlb
}
vrrp_sync_groups VG1 {
group {
VI_1
}
}
#Please change “ens160” to the interface name on you loadbalancer hosts.
#In some case it will be eth0, ens16xxx etc.
vrrp_instance VI_1 {
interface ens32

track_interface {
ens32
}

state BACKUP
virtual_router_id 51
priority 8

virtual_ipaddress {
172.16.251.136/32
}
advert_int 1
authentication {
auth_type PASS
auth_pass d0cker
}

}
########## Section for Harbor use HTTP protocol ######################
#Please change , , to real ip address
#virtual_server 80 {
# delay_loop 15
# lb_algo rr
# lb_kind DR
# protocol TCP
# nat_mask 255.255.255.0
# persistence_timeout 10
#
# real_server 80 {
# weight 10
# MISC_CHECK {
# misc_path “/usr/local/bin/check.sh “
# misc_timeout 5
# }
# }
#
# real_server 80 {
# weight 10
# MISC_CHECK {
# misc_path “/usr/local/bin/check.sh “
# misc_timeout 5
# }
# }
#}

###################End of HTTP

####################HTTPS#################################
#Please uncomment the follow when harbor running under https
virtual_server 172.16.251.136 443 {
delay_loop 15
lb_algo rr
lb_kind DR
protocol TCP
nat_mask 255.255.255.0
persistence_timeout 10

real_server 172.16.251.131 443 {
weight 10
MISC_CHECK {
misc_path “/usr/local/bin/check.sh 172.16.251.131”
misc_timeout 5
}
}

real_server 172.16.251.132 443 {
weight 10
MISC_CHECK {
misc_path “/usr/local/bin/check.sh 172.16.251.132”
misc_timeout 5
}
}
}
#########################End of HTTPS Section#################

root@Userver05:~# cat /usr/local/bin/check.sh
#!/bin/bash

set -e
#get protocol

LOG=/var/log/keepalived_check.log
nodeip=$1
nodeaddress=”http://${nodeip}“
http_code=curl -s -o /dev/null -w "%{http_code}" ${nodeaddress}

if [ $http_code == 200 ] ; then
protocol=”http”
elif [ $http_code == 301 ]
then
protocol=”https”
else
# echo “date +"%Y-%m-%d %H:%M:%S" $1, CHECK_CODE=$http_code” >> $LOG
exit 1
fi

systeminfo=curl -k -o - -s ${protocol}://${nodeip}/api/systeminfo

echo $systeminfo | grep “registry_url”
if [ $? != 0 ] ; then
exit 1
fi
#TODO need to check Clair, but currently Clair status api is unreachable from LB.
# echo $systeminfo | grep “with_clair” | grep “true”
# if [ $? == 0 ] ; then
# clair is enabled
# do some clair check
# else
# clair is disabled
# fi

check top api

http_code=curl -k -s -o /dev/null -w "%{http_code}\n" ${protocol}://${nodeip}/api/repositories/top
set +e
if [ $http_code == 200 ] ; then
exit 0
else
exit 1
fi

root@Userver04:~# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.251.136:https rr persistent 10
-> 172.16.251.131:https Route 10 2 0
-> 172.16.251.132:https Route 10 0 0
root@Userver04:~#

测试

root@Userver04:~# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.251.136:https rr persistent 10
-> 172.16.251.132:https Route 10 6 1
root@Userver04:~#