Pulsar 伪分布式部署

What can you get from this article:

• 3 nodes Zookeeper in pseudo cluster

• 3 nodes Pulsar in pesudo cluster(Optional change different znode)

  Prerequisites

• deploy enviroment

  • Mac Mini M1
  • hostname is macmini
  • JDK: 1.8.0_311
  • Deploy path: /Users/futeng/workspaces/binft/pulsar3nodes
• pulsar and zk
  • Pulsar version : 2.6.1
  • œœZookeeper version : 3.6.1

    Deploy 3 nodes Zoopkeeper pseudo-cluster

    Preview about how to delpoy Zookeeper:

1. Decompression Apache Zookeeper binary package.
2. Configure _zoo.cfg _with your own custom parameters.
3. Ship the first directory to another two nodes.
4. Start the zkServer.sh and test the cluster.

   Configure the first ZK node

   We need change different port to allow 3 ZK in single host. The first configuration like below: ```shell cat conf/zoo.cfg |grep -v “#” | grep -E “dataDir|clientPort|admin.enableServer|admin.serverPort|server.”

dataDir=/Users/futeng/workspaces/binft/pulsar3nodes/zk1/data clientPort=12181 admin.enableServer=true admin.serverPort=15678 server.1=macmini:12888:13888 server.2=macmini:22888:23888 server.3=macmini:32888:33888

<a name="dH87M"></a>
#### Ship to next 2 nodes
Copy to generante the next 2 nodes. The directory now looks below. 
```shell
.
├── zk1
├── zk2
└── zk3

We change the next 2 nodes using same rule.

# zk1
clientPort=12181
admin.serverPort=15678
# zk2
clientPort=22181
admin.serverPort=25678
# zk3
clientPort=32181
admin.serverPort=35678

Give each zk node myid.

echo "1" > zk1/data/myid
echo "2" > zk2/data/myid
echo "3" > zk3/data/myid

Now we know the zookeeper hierarchy connect address is : macmini:12181,macmini:22181,macmini:32181

Check all three nodes

➜  pulsar3nodes cat zk1/conf/zoo.cfg |grep -v "#" | grep -E "dataDir|clientPort|admin.enableServer|admin.serverPort|server."
dataDir=/Users/futeng/workspaces/binft/pulsar3nodes/zk1/data
clientPort=12181
admin.enableServer=true
admin.serverPort=15678
server.1=macmini:12888:13888
server.2=macmini:22888:23888
server.3=macmini:32888:33888
➜  pulsar3nodes cat zk2/conf/zoo.cfg |grep -v "#" | grep -E "dataDir|clientPort|admin.enableServer|admin.serverPort|server."
dataDir=/Users/futeng/workspaces/binft/pulsar3nodes/zk2/data
clientPort=22181
admin.enableServer=true
admin.serverPort=25678
server.1=macmini:12888:13889
server.2=macmini:22888:23889
server.3=macmini:32888:33889
➜  pulsar3nodes cat zk3/conf/zoo.cfg |grep -v "#" | grep -E "dataDir|clientPort|admin.enableServer|admin.serverPort|server."
dataDir=/Users/futeng/workspaces/binft/pulsar3nodes/zk3/data
clientPort=32181
admin.enableServer=true
admin.serverPort=35678
server.1=macmini:12888:13889
server.2=macmini:22888:23889
server.3=macmini:32888:33889

Start zk cluster and test it

Using shell script to boots faster.

# start zk
#!/bin/bash
zk1/bin/zkServer.sh start
zk2/bin/zkServer.sh start
zk3/bin/zkServer.sh start
zk1/bin/zkCli.sh -server macmini:12181,macmini:22181,macmini:32181
# stop zk
#!/bin/bash
zk1/bin/zkServer.sh stop
zk2/bin/zkServer.sh stop
zk3/bin/zkServer.sh stop

Deploy Pulsar in pseudo mode

Quick view:

1. Configure one pulsar node(broker and bookie).
2. Ship to the next two nodes
3. Initialize metadata
4. Start all broker and bookie service.
5. Test Pulsar using pulsar cli and pulsar-perf

Decompression Apache Pulsar binary package. Still we deploy in single host. So the first node could named like pulsar-1, ane the next two nodes will call pulsar-2 and pulsar-3 . All of them put in same directory.
We need prepare one Cluster name, and this moment I call it mini.
And for the clean of Zookeeper root path, I also choice another znode to store Pulsar metastore. More info in How to choice another znode when deploy pulsar.

Configure one pulsar node

Configure broker.confg

Tell the broker which cluster he belongs to.
Caution: Don’t changet bookkeeperMetadataServiceUri , still don’t know the reason.

cat pulsar-1/conf/broker.conf \
| grep -v "#" \
| grep -E "zookeeperServers|configurationStoreServers|clusterName|brokerServicePort|brokerServicePortTls|webServicePort|webServicePortTls|bookkeeperMetadataServiceUri|managedLedgerDefaultEnsembleSize|managedLedgerDefaultWriteQuorum|managedLedgerDefaultAckQuorum"
zookeeperServers=macmini:12181,macmini:22181,macmini:32181/pulsar-metadata-store
configurationStoreServers=macmini:12181,macmini:22181,macmini:32181/pulsar-configuration-metadata-store
brokerServicePort=16650
brokerServicePortTls=16651
webServicePort=18080
webServicePortTls=18443
clusterName=mini
managedLedgerDefaultEnsembleSize=2
managedLedgerDefaultWriteQuorum=2
managedLedgerDefaultAckQuorum=2

Config bookkeeper.conf

Tell the BookKeeper(also know as bookie) which ZK he report.

cat conf/bookkeeper.conf | grep -v "#" \
| grep -E "bookiePort|zkServers|zkLedgersRootPath|journalDirectory|ledgerDirectories"
bookiePort=13181
journalDirectory=data/bookkeeper/journal
ledgerDirectories=data/bookkeeper/ledgers
zkLedgersRootPath=/pulsar-metadata-store/ledgers
zkServers=macmini:12181,macmini:22181,macmini:32181

(optional) config log4j2

Open flush mode in log4j2.yamlfor log output, it’s optional but useful in dev enviroment.

immediateFlush: true

Ship to another

Simplly copy all to the next 2 nodes.

cp -r pulsar-1 pulsar-2
cp -r pulsar-1 pulsar-3

Different port in borker.conf

The core setting to deploy a pesude cluster is change different port.

# command to show different port
$ cat conf/broker.conf \
| grep -v "#" \
| grep -E "brokerServicePort|brokerServicePortTls|webServicePort|webServicePortTls"
# pulsar-1
brokerServicePort=16650
brokerServicePortTls=16651
webServicePort=18080
webServicePortTls=18443
# pulsar-2
brokerServicePort=26650
brokerServicePortTls=26651
webServicePort=28080
webServicePortTls=28443
# pular-3
brokerServicePort=36650
brokerServicePortTls=36651
webServicePort=38080
webServicePortTls=38443

Different port in bookkeeper.com

# Command to show 
$ cat pulsar-1/conf/bookkeeper.conf | grep -v "#" \
| grep -E "bookiePort|prometheusStatsHttpPort"
# pulsar-1
bookiePort=13181
prometheusStatsHttpPort=18000
# pulsar-2
bookiePort=23181
prometheusStatsHttpPort=28000
# pulsar-3
bookiePort=33181
prometheusStatsHttpPort=38000

Initialize metadata

Be sure keep the consitency of metadata. We all know the TLS service is not use, but please keep it in the initialize command for good ( can not execute if not).

pulsar-1/bin/pulsar initialize-cluster-metadata \
  --cluster mini \
  --zookeeper macmini:12181/pulsar-metadata-store \
  --configuration-store macmini:12181/pulsar-configuration-metadata-store \
  --web-service-url http://macmini:12181:18080 \
  --web-service-url-tls https://macmini:12181:18443 \
  --broker-service-url pulsar://macmini:12181:16650 \
  --broker-service-url-tls pulsar+ssl://macmini:12181:16651

You’ll know your cluster is ready by telled the init status from screen log, like Cluster metadata for 'mini' setup correctly.
You can list the zookeeper znode, and all we need is /pulsar-metadata-storefor now.

[zk: localhost:2181(CONNECTED) 2] ls /
[pulsar-configuration-metadata-store, pulsar-metadata-store, zookeeper]
[zk: localhost:2181(CONNECTED) 3] ls /pulsar-metadata-store
[bookies, ledgers, managed-ledgers, namespace, stream]

Start service

Start broker deamon

Start the broker foreground for the best practise, and it means good if you get the message like PulsarService started. Then start broker on daemon.

# start service on shell command, check it first
pulsar-1/bin/pulsar broker
# start service on daemon
pulsar-1/bin/pulsar-daemon start broker
# start all
$ cat broker-all-start.sh
#!/bin/bash
pulsar-1/bin/pulsar-daemon start broker
pulsar-2/bin/pulsar-daemon start broker
pulsar-3/bin/pulsar-daemon start broker
# stop all
cat broker-all-stop.sh
#!/bin/bash
pulsar-1/bin/pulsar-daemon stop broker
pulsar-2/bin/pulsar-daemon stop broker
pulsar-3/bin/pulsar-daemon stop broker

Start BookKeeper

Start the BookKeeper thread in foreground for test.

# foreground
pulsar-1/bin/pulsar bookie
# background
pulsar-1/bin/pulsar-daemon start bookie
# start all
cat bookie-all-start.sh
#!/bin/bash
pulsar-1/bin/pulsar-daemon start bookie
pulsar-2/bin/pulsar-daemon start bookie
pulsar-3/bin/pulsar-daemon start bookie
# stop all
$ cat bookie-all-stop.sh
#!/bin/bash
pulsar-1/bin/pulsar-daemon stop bookie
pulsar-2/bin/pulsar-daemon stop bookie
pulsar-3/bin/pulsar-daemon stop bookie

Test

Check jps

➜  pulsar3nodes jps
# three zk
4530 QuorumPeerMain
2798 QuorumPeerMain
2822 QuorumPeerMain
# three broker
5319 PulsarBrokerStarter
5533 PulsarBrokerStarter
5582 PulsarBrokerStarter
# three bookie
5690 BookieServer
5653 BookieServer
5622 BookieServer

Check metadata from Zookeeper

Get bookie info from ZK:

[zk: localhost:2181(CONNECTED) 181] ls /pulsar-metadata-store/ledgers/available
[192.1.216.107:3181, readonly]

Get broker info from ZK:

[zk: localhost:2181(CONNECTED) 184] ls /pulsar-metadata-store/namespace/pulsar/mini
[macmini:58080]
[zk: localhost:2181(CONNECTED) 186] get /pulsar-metadata-store/namespace/pulsar/mini/macmini:58080/0x00000000_0xffffffff
{"nativeUrl":"pulsar://macmini:56650","nativeUrlTls":"pulsar+ssl://macmini:56651","httpUrl":"http://macmini:58080","httpUrlTls":"https://macmini:58443","disabled":false,"advertisedListeners":{}}

Test with pulsar client

Config pulsar client

cat conf/client.conf | grep -v "#" \
| grep -E "webServiceUrl|brokerServiceUrl"
webServiceUrl=http://localhost:58080/
brokerServiceUrl=pulsar://localhost:56650/

Test with pub-sub

 pulsar-1/bin/pulsar-client consume \
  persistent://public/default/test \
  -n 100 \
  -s "consumer-test" \
  -t "Exclusive"
  pulsar-1/bin/pulsar-client produce \
  persistent://public/default/test \
  -n 1 \
  -f /etc/hosts

Check stats-internal

bin/pulsar-admin topics stats-internal \
  persistent://public/default/test
  {
  "entriesAddedCounter" : 3,
  "numberOfEntries" : 3,
  "totalSize" : 1392,
  "currentLedgerEntries" : 3,
  "currentLedgerSize" : 1392,
  "lastLedgerCreatedTimestamp" : "2022-03-17T10:16:15.201+08:00",
  "waitingCursorsCount" : 1,
  "pendingAddEntriesCount" : 0,
  "lastConfirmedEntry" : "0:2",
  "state" : "LedgerOpened",
  "ledgers" : [ {
    "ledgerId" : 0,
    "entries" : 0,
    "size" : 0,
    "offloaded" : false
  } ],
  "cursors" : {
    "consumer-test" : {
      "markDeletePosition" : "0:2",
      "readPosition" : "0:3",
      "waitingReadOp" : false,
      "pendingReadOps" : 0,
      "messagesConsumedCounter" : 3,
      "cursorLedger" : 1,
      "cursorLedgerLastEntry" : 3,
      "individuallyDeletedMessages" : "[]",
      "lastLedgerSwitchTimestamp" : "2022-03-17T10:16:15.266+08:00",
      "state" : "Open",
      "numberOfEntriesSinceFirstNotAckedMessage" : 1,
      "totalNonContiguousDeletedMessagesRange" : 0,
      "properties" : { }
    }
  }
}

Clean

Clean all data is alwasy in danger, but also help you quick reinstall your cluster.

clean metadata in Zookeeper

deleteall /counters
deleteall /loadbalance
deleteall /managed-ledgers
deleteall /namespace
deleteall /schemas
deleteall /admin
# may change znode
deleteall /pulsar-configuration-metadata-store
deleteall /pulsar-metadata-store

clean BookKeeper data

mv pulsar-1/data pulsar-1/databak
mv pulsar-2/data pulsar-2/databak
mv pulsar-3/data pulsar-3/databak

Others

1. first init, not admin znode created.

• published in mudium: https://medium.com/@ifuteng/pulsar-pseudo-cluster-deployment-fcb8214360ea