部署fabric于k8s之上 | Zhuang's Diary

hyperledger fabric 2.2 kubernetes

一、本地开发环境依赖

# Minikube 安装 v1.20.0
curl -Lo minikube https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v1.20.0/minikube-darwin-amd64
chmod +x minikube
sudo mv minikube /usr/local/bin/
# Minikube start
minikube start --image-mirror-country cn \
    --iso-url=https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/iso/minikube-v1.20.0.iso \
    --registry-mirror=https://xxxxx.mirror.aliyuncs.com

• 安装 kubectl 工具
• 安装 git
• 安装 docker
• 安装 hyperledger 相关命令列工具

# 执行以下指令下载 Hyperledger 命令列工具
curl -sSL http://raw.githubusercontent.com/hyperledger/fabric/master/scripts/bootstrap.sh | bash -s -- 2.2.0 1.5.0
# 或者
curl -sSL https://bit.ly/2ysbOFE | bash -s
# 设定环境变数至下载下来的 bin 目录下
export PATH=<path to download location>/bin:$PATH

以上脚本会在本地下载 k8s 部署需要的容器和可执行文件，云上的话需要准备的镜像如下：

centos:latest
hyperledger/fabric-orderer:2.2
hyperledger/fabric-peer:2.2.0
hyperledger/fabric-tools:2.2.0

用于测试的 chaincode 镜像，marbles 是 hyperledger 官方给的一个 sample，但是源码没有打镜像，这里其他人打的一个镜像，我们只是用来测试部署是否成功，之后会用我们开发的存证 chaincode 镜像替换

paragones/chaincode-marbles:1.0

# 创建一个名为 hyperledger 的 namespace
kubectl create namespace hyperledger
# 切换至 hyperledger namespace 
kubectl config set-context --current --namespace=hyperledger

二、系统架构规划

2.1 节点规划

下图为本范例的部署架构，所有 hyperledger 所需节点皆部署于 k8s 中的 hyperledger namespace 当中。架构中包含的节点如下

• orderer0 : 排序节点 0，用于排序区块
• orderer1 : 排序节点 1，用于排序区块
• orderer2 : 排序节点 2，用于排序区块
• peer0-org1 : 组织 1 的 Peer 节点，用于区块的实际运算、背书以及记帐。
• cli-org1-peer : 用于操纵组织 1 的 Peer 节点
• peer0-org2 : 组织 2 的 Peer 节点，用于区块的实际运算、背书以及记帐。
• cli-org2-peer : 用于操纵组织 2 的 Peer 节点

2.2 存储存放规划

三、准备凭证

Hyperledger Fabric 于节点沟通时必须依赖凭证进行沟通，因此必须先签发凭证。在凭证的签发过程中可以使用两种方式签发凭证

• cryptogen 命令
  cryptogen 为 Hyperledger Fabric 生成凭证的命令列工具，于 crypto-config.yaml 定义 Orderer 以及各组织的 Peer 的数量。
• fabric-ca 服务
  Fabric CA 是一个为 Hyperledger Fabric 签发凭证的工具，通常每个组织会有自己的 Fabric CA，通過 fabric-ca client 获得凭证后，就可以用这些凭证访问 Peer。

在此范例中将以 cryptogen 命令列生成各节点所需凭证，以下为签发凭证的指令。可以查看 git repo 中的 crypto-config.yaml 为签发凭证设定文件，crypto-config 则为签发结果。

cryptogen generate --config=crypto-config.yaml --output ./crypto-config

（notice：练习中，需要再次生成新的凭证设定文件时，需要将./crypto-config 文件夹删除后，再行运行上述命令。否则凭证设定文件不会被更新覆盖！)

四、准备创世区块

configtxgen -profile TwoOrgsOrdererGenesis -channelID devchan -outputBlock ./channel-artifacts/genesis.block

五、产生 Channel 所需档案

# 产生Channel 所需档案
configtxgen -profile TwoOrgsChannel -outputCreateChannelTx ./channel-artifacts/channel.tx -channelID "mychannel"
# 产生 Org1 使用的 channel 设定文件
configtxgen -profile TwoOrgsChannel -outputAnchorPeersUpdate ./channel-artifacts/org1Anchors.tx  -channelID "mychannel" -asOrg org1MSP
# 产生 Org2 使用的 channel 设定文件
configtxgen -profile TwoOrgsChannel -outputAnchorPeersUpdate ./channel-artifacts/org2Anchors.tx  -channelID "mychannel" -asOrg org2MSP

六、同步档案至 PV 中

在前述三、四、五章节中我们已经准备了以下内容

• 各节点所需凭证
• 创世区块
• Channel 设定档

由于 Hyperledger 之 container 必须先将以上档案放置至正确位置后 container 才能正常启动。但本次安装不使用 NFS 预先将档案填入，而是先启临时的 container 同步档案。
在本步骤我们将启动两个用于填充档案的 container

以下为同步文档的步骤

6.1 启动临时 container

kubectl create -f /deploy-hyperledger-fabric-on-k8s/file-populate-bastion/

这个指令将会产生 pv/pvc/orderer-bastion/peer-bastion 这些资源

6.2 同步 Orderer 所需文档

# 登录进入 orderer-bastion
kubectl exec -it ${orderer-bastion-pod-name} bash

# 安装 git 并 clone deploy-hyperledger-fabric-on-k8s repo
yum install git -y 
git clone https://github.com/willzhuang/deploy-hyperledger-fabric-on-k8s.git

# 同步 orderer0 创世区块
cp deploy-hyperledger-fabric-on-k8s/channel-artifacts/genesis.block /orderer0-pvc/
# 同步 orderer0 凭证
cp -r deploy-hyperledger-fabric-on-k8s/crypto-config/ordererOrganizations/consortium/orderers/orderer0/* /orderer0-pvc/

# 同步 orderer1 创世区块
cp deploy-hyperledger-fabric-on-k8s/channel-artifacts/genesis.block /orderer1-pvc/
# 同步 orderer1 凭证
cp -r deploy-hyperledger-fabric-on-k8s/crypto-config/ordererOrganizations/consortium/orderers/orderer1/* /orderer1-pvc/

# 同步 orderer2 创世区块
cp deploy-hyperledger-fabric-on-k8s/channel-artifacts/genesis.block /orderer2-pvc/
# 同步 orderer2 凭证
cp -r deploy-hyperledger-fabric-on-k8s/crypto-config/ordererOrganizations/consortium/orderers/orderer2/* /orderer2-pvc/

6.3 同步 Peer 所需要的文档

# 登录进入 peer-bastion
kubectl exec -it ${peer-bastion-pod-name} bash

# 安装 git 並 clone deploy-hyperledger-fabric-on-k8s repo
yum install git -y 
git clone https://github.com/willzhuang/deploy-hyperledger-fabric-on-k8s.git

# 同步 org1 peer0 所需之凭证
cp -r /deploy-hyperledger-fabric-on-k8s/crypto-config/peerOrganizations/org1/peers/peer0-org1/* peer0-org1-pvc/
# 同步 org1 peer0 cli 所需之凭证
cp -r /deploy-hyperledger-fabric-on-k8s/crypto-config/* /peer0-org1-pvc/
# 同步 org1 peer0 所需之 channel设定文档
cp /deploy-hyperledger-fabric-on-k8s/channel-artifacts/* peer0-org1-pvc/

# 同步 org2 peer0 所需之凭证
cp -r /deploy-hyperledger-fabric-on-k8s/crypto-config/peerOrganizations/org2/peers/peer0-org2/* peer0-org2-pvc/
# 同步 org2 peer0 cli 所需之凭证
cp -r /deploy-hyperledger-fabric-on-k8s/crypto-config/* /peer0-org2-pvc/
# 同步 org2 peer0 所需之 channel设定文档
cp /deploy-hyperledger-fabric-on-k8s/channel-artifacts/* peer0-org2-pvc/

七、启动 Orderer

# 启动 orderer cluster
kubectl create -f /deploy-hyperledger-fabric-on-k8s/orderer/

內含 orderer0,orderer1,orderer2 所需的 deployment 与 service 资源

八、启动 Peer

# 启动 org1 peer0 
kubectl create -f /deploy-hyperledger-fabric-on-k8s/org1/
# 內含 org1 peer0/cli 所需的 deployment,configmap,service 资源

# 启动 org2 peer0 
kubectl create -f /deploy-hyperledger-fabric-on-k8s/org2/
# 內含 org2 peer0/cli 所需的 deployment,configmap,service 资源

九、创建 channel

9.1 将 org1 peer0 加入 channel

# 登录进入 cli pod 
kubectl exec -it ${org1-peer0-cli-pod-name} sh

# 产生 channel 区块
peer channel create -o orderer0:7050 -c mychannel -f ./scripts/channel-artifacts/channel.tx --tls true --cafile $ORDERER_CA

运行成功后，log 信息如下：

/opt/gopath/src/github.com/hyperledger/fabric/peer # peer channel create -o orderer0:7050 -c mychannel -f ./scripts/channel-artifacts/channel.tx -
-tls true --cafile $ORDERER_CA
2021-08-10 05:41:03.212 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2021-08-10 05:41:03.242 UTC [cli.common] readBlock -> INFO 002 Expect block, but got status: &{NOT_FOUND}
2021-08-10 05:41:03.246 UTC [channelCmd] InitCmdFactory -> INFO 003 Endorser and orderer connections initialized
2021-08-10 05:41:03.447 UTC [cli.common] readBlock -> INFO 004 Expect block, but got status: &{SERVICE_UNAVAILABLE}
2021-08-10 05:41:03.450 UTC [channelCmd] InitCmdFactory -> INFO 005 Endorser and orderer connections initialized
2021-08-10 05:41:03.651 UTC [cli.common] readBlock -> INFO 006 Expect block, but got status: &{SERVICE_UNAVAILABLE}
2021-08-10 05:41:03.654 UTC [channelCmd] InitCmdFactory -> INFO 007 Endorser and orderer connections initialized
2021-08-10 05:41:03.857 UTC [cli.common] readBlock -> INFO 008 Expect block, but got status: &{SERVICE_UNAVAILABLE}
2021-08-10 05:41:03.866 UTC [channelCmd] InitCmdFactory -> INFO 009 Endorser and orderer connections initialized
2021-08-10 05:41:04.068 UTC [cli.common] readBlock -> INFO 00a Expect block, but got status: &{SERVICE_UNAVAILABLE}
2021-08-10 05:41:04.070 UTC [channelCmd] InitCmdFactory -> INFO 00b Endorser and orderer connections initialized
2021-08-10 05:41:04.273 UTC [cli.common] readBlock -> INFO 00c Expect block, but got status: &{SERVICE_UNAVAILABLE}
2021-08-10 05:41:04.277 UTC [channelCmd] InitCmdFactory -> INFO 00d Endorser and orderer connections initialized
2021-08-10 05:41:04.480 UTC [cli.common] readBlock -> INFO 00e Received block: 0

# 加入 channel
peer channel join -b mychannel.block

运行成功后，log 信息如下：

/opt/gopath/src/github.com/hyperledger/fabric/peer # peer channel join -b mychannel.block
2021-08-10 05:48:59.716 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2021-08-10 05:48:59.736 UTC [channelCmd] executeJoin -> INFO 002 Successfully submitted proposal to join channel

# 查看是否在 channel 当中
peer channel list

9.2 將 org2 peer0 加入 channel

# 登录进入 cli pod 
kubectl exec -it pod ${org2-peer0-cli-pod-name} sh

# 取得 channel 区块
peer channel fetch 0 mychannel.block -c mychannel -o orderer0:7050 --tls --cafile $ORDERER_CA

# 加入 channel
peer channel join -b mychannel.block

# 查看是否在 channel 当中
peer channel list

运行成功后，log 信息如下：

/opt/gopath/src/github.com/hyperledger/fabric/peer # peer channel fetch 0 mychannel.block -c mychannel -o orderer0:7050 --tls --cafile $ORDERER_CA
2021-08-10 06:05:04.550 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2021-08-10 06:05:04.553 UTC [cli.common] readBlock -> INFO 002 Received block: 0
/opt/gopath/src/github.com/hyperledger/fabric/peer # peer channel join -b mychannel.block
2021-08-10 06:05:16.493 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2021-08-10 06:05:16.513 UTC [channelCmd] executeJoin -> INFO 002 Successfully submitted proposal to join channel
/opt/gopath/src/github.com/hyperledger/fabric/peer # peer channel list
2021-08-10 06:05:24.222 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
Channels peers has joined: 
mychannel

十、Fabric 节点上安装外的 Chaincode

10.1 安装 chaincode
‘marbles’ 链码作为范例

打包 org1

# 进入github的chaincode/packaging
# 将connection.json 打包成为 code.tar.gz
$ tar cfz code.tar.gz connection.json

#再將code.tar.gz metadata.json包成marbles-org1.tgz
$ tar cfz marbles-org1.tgz code.tar.gz metadata.json

将 org1 tar 文档安装到 peer cli pod 中

# 将marbles-org1.tgz放入peer pod
kubectl cp marbles-org1.tgz hyperledger/${cli-org1-name}:/opt/gopath/src/github.com/hyperledger/fabric/peer

# 登录进入${cli-org1-name} pod
kubectl exec -it ${cli-org1-name} -- /bin/bash

# 在peer-cli上安装chaincode
$ peer lifecycle chaincode install marbles-org1.tgz

# 查询链码，识别字符串
$ peer lifecycle chaincode queryinstalled

打包 org2，如打包 or1 一样的步骤，请修改 connection.json 中的 address

"address": "chaincode-marbles-org2.hyperledger:7052"

依照下列打包步骤

$ rm -f code.tar.gz
$ tar cfz code.tar.gz connection.json
$ tar cfz marbles-org2.tgz code.tar.gz metadata.json
$ peer lifecycle chaincode install marbles-org2.tgz

# 查询链码，识别字符串
$ peer lifecycle chaincode queryinstalled

至此，已完成外部链码设置。

10.2 部署” 智能合约”

制作 golang 的 alpine 镜像文件

$ docker build -t chaincode/marbles:1.0 .

# 将文件推送至dockerhub
$ docker login
$ docker push chaincode/marbles:1.0

进入 github 中 chaincode\k8s ，找到 org1-chaincode-deployment.yaml 和 org2-chaincode-deployment.yaml 中的 CHAINCODE_CCID

将 yaml 文件中的 CHAINCODE_CCID 更换成为对应的 chaincode 识别码

# 部署 chaincode 到aks上
$ kubectl create -f chaincode/k8s

完成部署。

10.3 审核链码的安装

请在 peer-cli-org1/peer-cli-org2 这 2 个 pod 中允许链码的安装，记得修改 CHAINCODE_CCID
※peer-cli-org1 及 peer-cli-org2 的 CHAINCODE_CCID 需要对应到各自的 chaincode 识别码

$ peer lifecycle chaincode approveformyorg --channelID mychannel --name marbles --version 1.0 --init-required --package-id marbles:e001937433673b11673d660d142c722fc372905db87f88d2448eee42c9c63064 --sequence 1 -o orderer0:7050 --tls --cafile $ORDERER_CA --signature-policy "AND ('org1MSP.peer','org2MSP.peer')

检查所有 org 允许状态

$peer lifecycle chaincode checkcommitreadiness --channelID mychannel --name marbles --version 1.0 --init-require
d --sequence 1 -o -orderer0:7050 --tls --cafile $ORDERER_CA --signature-policy "AND ('org1MSP.peer','org2MSP.peer')"

至此，2 个 peer-cli 都审核完毕。

之后提交到 channel 中

peer lifecycle chaincode commit -o orderer0:7050 --channelID mychannel --name marbles --version 1.0 --sequence 1 --init-required --tls true --cafile $ORDERER_CA --peerAddresses peer0-org1:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0-org1/tls/ca.crt --peerAddresses peer0-org2:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2/peers/peer0-org2/tls/ca.crt --signature-policy "AND ('org1MSP.peer','org2MSP.peer')"

10.4 测试

# 查询安装的链码
peer lifecycle chaincode queryinstalled
# 查询审核通过的链码
peer lifecycle chaincode queryapproved --channelID mychannel --name marbles
# 查询提交完成的链码
peer lifecycle chaincode querycommitted --channelID mychannel --name marbles
# 塞弹珠(第一次)
peer chaincode invoke -o orderer0:7050 --isInit --tls true --cafile $ORDERER_CA -C mychannel -n marbles \
--peerAddresses peer0-org1:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0-org1/tls/ca.crt  \
--peerAddresses peer0-org2:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2/peers/peer0-org2/tls/ca.crt  \
-c '{"Args":["initMarble","marble1","blue","35","tom"]}' --waitForEvent
# 塞弹珠
peer chaincode invoke -o orderer0:7050 --tls true --cafile $ORDERER_CA -C mychannel -n marbles \
--peerAddresses peer0-org1:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0-org1/tls/ca.crt \
--peerAddresses peer0-org2:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2/peers/peer0-org2/tls/ca.crt \
-c '{"Args":["initMarble","marble5","red","50","tom"]}' --waitForEvent
# 查询弹珠
peer chaincode query -C mychannel -n marbles -c '{"Args":["readMarble","marble5"]}'

开发 Client Application

目前 hyperledger 不对外，请使用 kubectl port-forward 的方式连进去。
指令:

kubectl port-forward ${pord-id} 7050:7050

参考 fabric-samples 中 typescript 所撰写的范例 ※另有 java， go 语言的范例!!

以下为 typerscript 范例:

async function main() {
    try {
        const ccpPath = path.resolve(__dirname, '..', 'connection.json');

        const fileExists = fs.existsSync(ccpPath);
        if (!fileExists) {
            throw new Error('no such file or directory: ${ccpPath}');
        }
        const contents = fs.readFileSync(ccpPath, 'utf8');
        console.log(contents);

        const connectionProfile = JSON.parse(contents);
        const gateway = new Gateway();
        const wallet = await buildWallet(walletPath);

        const identity: X509Identity = {
            credentials: {
                //crypto-config\peerOrganizations\org1\users\Admin@org1\msp\signcerts\Admin@org1-cert.pem
                certificate: '-----BEGIN CERTIFICATE-----\nMIICBjCCAaygAwIBAgIRAKeXh5QGRlhFCf31amakNiowCgYIKoZIzj0EAwIwWzEL
                //crypto-config\peerOrganizations\org1\users\Admin@org1\msp\keystore
                privateKey: '-----BEGIN PRIVATE KEY-----\nMIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQg9tkGBGl8aIWYbN/i\nqKjl5dK3D3qXx6ltn3YrlH4NFPehRANCAATMJIf0mY3FQDysZOevzbsBwqttfVuW\nwoIFV9Z0TwBEXXAQ0HPd9u77zV4my0onty3rJnWKF2kuhIn5PUzz61zk\n-----END PRIVATE KEY-----\n',
            },
            mspId: org1UserId,
            type: 'X.509',
        };
        await wallet.put(org1UserId, identity);


        const gatewayOpts: GatewayOptions = {
            wallet,
            identity: org1UserId,
            discovery: { enabled: false, asLocalhost: false }, // using asLocalhost as this gateway is using a fabric network deployed locally
        };

        await gateway.connect(connectionProfile, gatewayOpts);

        const network = await gateway.getNetwork('mychannel');

        const contract = network.getContract('marbles');
        let result = await contract.evaluateTransaction('readMarble', 'marble2');
        console.log(`*** Result: ${prettyJSONString(result.toString())}`);
    } catch (error) {
        console.error(`******** FAILED to run the application: ${error}`);
    }
}

main();

上述为 app.ts，此 app.ts 会读取 connection.json 档，此档用于定义 peer url 以及 peer 的 tls 凭证路径。

{
    "name": "Network",
    "version": "1.1",
    "channels": {
        "mychannel": {
            "peers": [
                "peer0-org1",
                "peer0-org2"
            ]
        }
    },
    "organizations": {
        "Org1": {
            "mspid": "org1MSP",
            "peers": [
                "peer0-org1"
            ]
        },
        "Org2": {
            "mspid": "org2MSP",
            "peers": [
                "peer0-org2"
            ]
        }
    },
    "peers": {
        "peer0-org1": {
            "url": "grpcs://peer0-org1:7051",
            "grpcOptions": {
                "ssl-target-name-override": "peer0-org1"
            },
            "tlsCACerts": {
                "path": "crypto-config\peerOrganizations\org1\tlsca\tlsca.org1-cert.pem"
            }
        },
        "peer0-org2": {
            "url": "grpcs://peer0-org2:7051",
            "grpcOptions": {
                "ssl-target-name-override": "peer0-org2"
            },
            "tlsCACerts": {
                "path": "crypto-config\peerOrganizations\org2\tlsca\tlsca.org2-cert.pem"
            }
        }
    }
}

• application 连到 peer 的时候必须要有 tls 凭证去验证 peer service 是否安全及正确。
• 当交易的时候需要提供该使用者 (user or admin) 的交易凭证(x509 cert +private key)

路径对应如下：
tls CA Cert： crypto-config\peerOrganizations\org1\tlsca\tlsca.org1-cert.pem
X509 certificate： crypto-config\peerOrganizations\org1\users\Admin@org1\msp\signcerts\Admin@org1-cert.pem
X509 privateKey： crypto-config\peerOrganizations\org1\users\Admin@org1\msp\keystore\priv_sk
[color=#d85887]

成功收到资料如图

附录、参考连结

1. Hyperledger 参考链接

• Hyperledger 官方文件
• Hyperledger Git Repository
• Hyperledger 中文文档
• Hyperledger 手动部署全纪录
• 部署 Hyperledger2.2 于 Kubenetes
• 部署 Hyperledger2.2 于 Kubenetes 代码
• 凭证说明

1. Kubenetes 参考链接

• k8s 佈署用指令參考

1. NFS 架构参考链接

• NFS 架构参考

1. 部署 fabric 和 CA 与 Kubenetes 之上
   Deploy Hyperledger Fabric network on Kubernetes
   https://willzhuang.github.io/2021/08/09/%E9%83%A8%E7%BD%B2fabric%E4%BA%8Ek8s%E4%B9%8B%E4%B8%8A/